DEV Community: Ranjan Purbey

Background Jobs in SvelteKit with BullMQ

Ranjan Purbey — Sat, 25 Jan 2025 22:24:43 +0000

Is your SvelteKit app struggling with tasks like sending emails, resizing images, or processing data? With BullMQ, you can offload these heavy jobs to the background and keep your app lightning-fast. In this post, we’ll show you how to set it up and tackle real-world tasks like a pro. Let’s dive in!

tl;dr Set up BullMQ workers in hooks.server.js. Check out the example

What is BullMQ?

BullMQ is a Node.js library for creating and managing job queues with Redis. It helps you run time-consuming tasks in the background efficiently. With built-in features like retries, job scheduling, and concurrency control, BullMQ makes handling complex workflows in your app simple and reliable.

Step 1: Install dependencies

First, install ioredis (Redis client for node.js) and bullmq:

pnpm i -D ioredis bullmq

Even though you can add jobs to a bullmq queue from a serverless environment like Vercel, the workers must run on a traditional long-lived node.js server. Hence, replace adapter-auto with adapter-node:

pnpm rm @sveltejs/adapter-auto && pnpm i -D @sveltejs/adapter-node

Don't forget to update your Svelte config (svelte.config.js) with the newly installed node adapter.

Step 2: Setup Job Queue and Processor

Next, let's set up a BullMQ job queue and its processor. Create a .js file in the src/lib/server/ directory:

// src/lib/server/background-jobs.js

import { REDIS_URL } from "$env/static/private";
import { Queue, Worker } from "bullmq";
import IORedis from "ioredis";

const Q_NAME = "q";

export const jobsQueue = new Queue(Q_NAME, {
  connection: new IORedis(REDIS_URL),
});

const sleep = (t) => new Promise((resolve) => setTimeout(resolve, t * 100));

export const setupBullMQProcessor = () => {
  new Worker(
    Q_NAME,
    async (job) => {
      for (let i = 0; i <= 100; i++) {
        await sleep(Math.random());
        await job.updateProgress(i);
        await job.log(`Processing job at interval ${i}`);

        if (Math.random() * 200 < 1) throw new Error(`Random error at ${i}`);
      }

      return `This is the return value of job (${job.id})`;
    },
    // https://docs.bullmq.io/bull/patterns/persistent-connections#maxretriesperrequest
    { connection: new IORedis(REDIS_URL, { maxRetriesPerRequest: null }) }
  );
};

Here, we've also created a utility function that instantiates a BullMQ worker to listen for and process jobs in the queue Q_NAME.

We need to call this function in our hooks.server.js file—either at the top level or within the init hook.

// src/hooks.server.js

// ...
import { building } from "$app/environment";
import { setupBullMQProcessor } from "$lib/server/background-jobs";
// ...
if (!building) {
  setupBullMQProcessor();
}
// ...

The !building check skips setting up the worker (and in turn a Redis connection) during the build, speeding up the process.

🎉 BullMQ is now ready to be used in our SvelteKit app 🎉

Demo time

In order to test the setup, let's create a POST endpoint to enqueue a job.

// src/routes/+server.ts

import { jobsQueue } from "$lib/server/background-jobs";

export const POST = async () => {
  const { id: jobId } = await jobsQueue.add("job", {});

  /*
  The following code passes the job's progress to the client as a stream.
  If you don't need to update the client with the progress, you can skip
  the following. You can also use web-sockets or polling for that.
  */
  const stream = new ReadableStream({
    async pull(controller) {
      const job = await jobsQueue.getJob(jobId);
      controller.enqueue(
        JSON.stringify(
          job.failedReason
            ? { error: job.failedReason }
            : job.returnvalue
            ? { data: job.returnvalue }
            : { progress: job.progress }
        )
      );
      controller.enqueue("\n");

      if (job.finishedOn) {
        controller.close();
      }

      // wait for 1-second before sending the next status update
      await new Promise((r) => setTimeout(r, 1e3));
    },
  });

  return new Response(stream, {
    headers: { "content-type": "text/plain" },
  });
};

And on the frontend, let's add a button to trigger the above endpoint and subsequently show the job's status:

<!-- src/routes/+page.svelte -->

<script>
  let result = $state();

  $inspect(result);

  const handleClick = async () => {
    const response = await fetch("/", { method: "post" });
    const reader = await response.body.getReader();
    while (true) {
      const { done, value } = await reader.read();
      if (done) break;
      result = JSON.parse(new TextDecoder().decode(value));
    }

    setTimeout(() => (result = undefined), 3e3);
  };
</script>

{#if result?.error}
  <div class="error">{result.error}</div>
{:else if result?.data}
  <div class="success">{result.data}</div>
{:else if result?.progress !== undefined}
  <label>
    {result.progress === 100 ? "Done" : "Processing..."}
    <progress value={result.progress} max="100"></progress>
    {result.progress}%
  </label>
{:else}
  <button onclick={handleClick}>Schedule Background Job</button>
{/if}

<style>
  .error {
    color: red;
  }
  .success {
    color: darkgreen;
  }
</style>

Here is the output:

Bonus 🎁

You can also mount a bull-board dashboard in your SvelteKit app for easy monitoring of background jobs.

Install bull-board dependencies

pnpm i -D @bull-board/api @bull-board/hono @hono/node-server hono

and modify your hooks.server.js:

// src/hooks.server.js

import { building } from "$app/environment";
import { jobsQueue, setupBullMQProcessor } from "$lib/server/background-jobs";
import { createBullBoard } from "@bull-board/api";
import { BullMQAdapter } from "@bull-board/api/bullMQAdapter";
import { HonoAdapter } from "@bull-board/hono";
import { serveStatic } from "@hono/node-server/serve-static";
import { Hono } from "hono";

if (!building) {
  setupBullMQProcessor();
}

const bullboard = (() => {
  const serverAdapter = new HonoAdapter(serveStatic);

  createBullBoard({
    queues: [new BullMQAdapter(jobsQueue)],
    serverAdapter,
  });
  const app = new Hono({ strict: false });
  const basePath = "/jobs";
  serverAdapter.setBasePath(basePath);
  app.route(basePath, serverAdapter.registerPlugin());

  return app;
})();

export const handle = async ({ event, resolve }) => {
  if (event.url.pathname.match(/^\/jobs($|\/)/)) {
    return bullboard.fetch(event.request);
  }

  return resolve(event);
};

Then visit <YOUR_SERVER_URL>/jobs to see the bull-board dashboard

Full example:

ranjan-purbey / sveltekit-bullmq-demo

BullMQ with SvelteKit

Simple example to set up background jobs in a SvelteKit app using BullMQ

Read detailed explanation here

Usage

Clone this repo
Install dependencies
```
pnpm i
```
Add REDIS_URL in your .env file.
Start the dev server
```
pnpm dev
```
or build and preview
```
pnpm build && pnpm preview
```

View on GitHub

Use Tailwind CSS v2.0 with Rails

Ranjan Purbey — Sat, 23 Jan 2021 19:26:56 +0000

Recently, the folks behind Tailwind CSS released v2.0 🎉🎉
(Well, that was in November 2020, but hey did time even exist in 2020?!)

Checkout the release notes

But unfortunately when you try to add it to your Rails project, Webpack throws the following error (I am assuming you have already setup @rails/webpacker in your project):

Error: PostCSS plugin tailwindcss requires PostCSS 8.

The error message is simply saying that Tailwind v2 depends on PostCSS 8, however the current builds of webpacker use PostCSS 7. To fix this issue we can take one of two approaches:

Easy Method

Well, Tailwind maintainers realized that PostCSS 8, being a recent release itself, is not very widely supported by popular frameworks like Rails. So, they were kind enough to release a PostCSS 7 compatible build for the time being.

All you have to do is

yarn remove tailwindcss
yarn add tailwindcss@npm:@tailwindcss/postcss7-compat

...aaand done!!

The compatibility build supports all of the features as that of the "normal" build. Awesome! Except for one catch though:

If you are working on a decently-sized project, the chances of the Rails version in your application being upgraded anytime soon may be low. And it is probable that the subsequent releases of Tailwind might not come with a compatibility build. So what do we do?

(not-so) Hard Method

Well, won't it be better if we could just bump the PostCSS version in our Rails app? That way we won't have to depend on or worry about a compatibility build.
Let's try

yarn add tailwindcss@^2.0.2 postcss@^8.2.4
./bin/webpack-dev-server

Hmm, same error message? Oh yes, we should also upgrade postcss-loader

yarn add postcss-loader@^4.2.0
./bin/webpack-dev-server

Another error message!!

ValidationError: Invalid options object. PostCSS Loader has been initialized using an options object that does not match the API schema.
 - options has an unknown property 'config'. These properties are valid:
   object { postcssOptions?, execute?, sourceMap?, implementation? }

Here webpack is complaining that postcss-loader was initialized with an invalid option config. In older versions of the loader, config was used to pass PostCSS configuration option. From the error message above, you can probably guess that it has been renamed to postcssOptions. "Dang! I have to edit webpack config now!" was my reaction, but turns out it's not that difficult.
If you look at the contents of your config/webpack/environment.js file, it will look something like this:

const { environment } = require('@rails/webpacker')

module.exports = environment

We change it to remove the config key from postcss-loader options for each file type (css, sass, moduleCss, etc.):

const { environment } = require("@rails/webpacker");

environment.loaders
  .map((rule) => rule.value.use)
  .flat()
  .filter(({ loader }) => loader === "postcss-loader")
  .forEach((item) => {
    if (item.options) {
      item.options.postcssOptions = { from: item.options.config.path };
      delete item.options.config;
    }
  });

module.exports = environment;

Done!! Now webpack should compile your bundle just fine.