How To Handle Asynchronous Tasks with Node.js and BullMQ

Web applications have request/response cycles. When you visit a URL, the browser sends a request to the server running an app that processes data or runs queries in the database. As this happens, the user is kept waiting until the app returns a response. For some tasks, the user can get a response quickly; for time-intensive tasks, such as processing images, analyzing data, generating reports, or sending emails, these tasks take a long time to finish and can slow down the request/response cycle.

Challenges of CPU-Intensive Tasks

For example, suppose you have an application where users upload images. In that case, you might need to resize, compress, or convert the image to another format to preserve your server’s disk space before showing the image to the user. Processing an image is a CPU-intensive task, which can block a Node.js thread until the task is finished. That might take a few seconds or minutes. Users have to wait for the task to finish to get a response from the server.

Introducing BullMQ for Background Tasks

To avoid slowing down the request/response cycle, you can use BullMQ, a distributed task (job) queue that allows you to offload time-consuming tasks from your Node.js app to BullMQ, freeing up the request/response cycle. This tool enables your app to send responses to the user quickly while BullMQ executes the tasks asynchronously in the background and independently from your app. To keep track of jobs, BullMQ uses Redis to store a short description of each job in a queue. A BullMQ worker then dequeues and executes each job in the queue, marking it complete once done.

Using BullMQ to Enhance Application Responsiveness

In this article, you will use BullMQ to offload a time-consuming task into the background, which will enable an application to respond quickly to users. First, you will create an app with a time-consuming task without using BullMQ. Then, you will use BullMQ to execute the task asynchronously. Finally, you will install a visual dashboard to manage BullMQ jobs in a Redis queue.

Prerequisites

• Node.js development environment set up.
• Redis installed on your system.
• Familiarity with promises and async/await functions.
• Basic knowledge of how to use Express.
• Familiarity with Embedded JavaScript (EJS).
• Basic understanding of how to process images with Sharp.

Step 1: Setting Up the Project Directory

In this step, you will create a directory and install the necessary dependencies for your application. The application you’ll build in this tutorial will allow users to upload an image, which is then processed using the Sharp package. Image processing is time-intensive and can slow the request/response cycle, making the task a good candidate for BullMQ to offload into the background. The technique you will use to offload the task will also work for other time-intensive tasks.

To begin, create a directory called image_processor and navigate into the directory:

mkdir image_processor && cd image_processor

Then, initialize the directory as an npm package:

npm init -y

The command creates a package.json file. The -y option tells npm to accept all the defaults.

Upon running the command, your output will match the following:

{
  "name": "image_processor",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}

The output confirms that the package.json file has been created. Important properties include the name of your app (name), your application version number (version), and the starting point of your project (main). If you want to learn more about the other properties, you can review npm’s package.json documentation.

The application you will build in this tutorial will require the following dependencies:

• express: a web framework for building web apps.
• express-fileupload: a middleware that allows your forms to upload files.
• sharp: an image processing library.
• ejs: a template language that allows you to generate HTML markup with Node.js.
• bullmq: a distributed task queue.
• bull-board: a dashboard that builds upon BullMQ and displays the status of the jobs with a nice User Interface (UI).

To install all these dependencies, run the following command:

npm install express express-fileupload sharp ejs bullmq @bull-board/express

In addition to the dependencies you installed, you will also use the following image later in this tutorial:

An image underwater with a ray of light coming into it

Use curl to download the image to the location of your choice on your local computer:

curl -O https://deved-images.nyc3.cdn.digitaloceanspaces.com/CART-68886/underwater.png

You have the necessary dependencies to build a Node.js app that does not have BullMQ, which you will do next.

Step 2: Implementing a Time-Intensive Task Without BullMQ

In this step, you will build an application with Express that allows users to upload images. The app will start a time-intensive task using Sharp to resize the image into multiple sizes, which are then displayed to the user after a response is sent. This step will help you understand how time-intensive tasks affect the request/response cycle.

Using nano, or your preferred text editor, create the index.js file:

nano index.js

In your index.js file, add the following code to import dependencies:

const path = require("path");
const fs = require("fs");
const express = require("express");
const bodyParser = require("body-parser");
const sharp = require("sharp");
const fileUpload = require("express-fileupload");

In the first line, you import the path module for computing file paths with Node. In the second line, you import the fs module for interacting with directories. You then import the express web framework. You import the body-parser module to add middleware to parse data in HTTP requests. Following that, you import the sharp module for image processing. Finally, you import express-fileupload for handling uploads from an HTML form.

Adding Middleware and Creating a Route

Next, add the following code to implement middleware in your app:

const app = express();
app.set("view engine", "ejs");
app.use(bodyParser.json());
app.use(
  bodyParser.urlencoded({
    extended: true,
  })
);

First, you set the app variable to an instance of Express. Second, using the app variable, the set() method configures Express to use the ejs template language. You then add the body-parser module middleware with the use() method to transform JSON data in HTTP requests into variables that can be accessed with JavaScript. In the following line, you do the same with URL-encoded input.

Next, add the following lines to add more middleware to handle file uploads and serve static files:

app.use(fileUpload());
app.use(express.static("public"));

You add middleware to parse uploaded files by calling the fileUpload() method, and you set a directory where Express will look at and serve static files, such as images and CSS.

With the middleware set, create a route that displays an HTML form for uploading an image:

app.get("/", function (req, res) {
  res.render("form");
});

Here, you use the get() method of the Express module to specify the / route and the callback that should run when the user visits the homepage or / route. In the callback, you invoke res.render() to render the form.ejs file in the views directory. You have not yet created the form.ejs file or the views directory.

To create it, first, save and close your file. In your terminal, enter the following command to create the views directory in your project root directory:

mkdir views

Move into the views directory:

cd views

Create the form.ejs file in your editor:

nano form.ejs

The form.ejs file and further details for handling uploads continue with similar instructions.

Step 3: Executing Time-Intensive Tasks Asynchronously with BullMQ

In this step, you will offload a time-intensive task to the background using BullMQ. This adjustment will free the request/response cycle and allow your app to respond to users immediately while the image is being processed.

To do that, you need to create a succinct description of the job and add it to a queue with BullMQ. A queue is a data structure that works similarly to how a queue works in real life. When people line up to enter a space, the first person on the line will be the first person to enter the space. Anyone who comes later will line up at the end of the line and will enter the space after everyone who precedes them in line until the last person enters the space.

With the queue data structure’s First-In, First-Out (FIFO) process, the first item added to the queue is the first item to be removed (dequeue). With BullMQ, a producer will add a job in a queue, and a consumer (or worker) will remove a job from the queue and execute it.

The queue in BullMQ is in Redis. When you describe a job and add it to the queue, an entry for the job is created in a Redis queue. A job description can be a string or an object with properties that contain minimal data or references to the data that will allow BullMQ to execute the job later. Once you define the functionality to add jobs to the queue, you move the time-intensive code into a separate function. Later, BullMQ will call this function with the data you stored in the queue when the job is dequeued. Once the task has finished, BullMQ will mark it completed, pull another job from the queue, and execute it.

Open index.js in your editor:

nano index.js

In your index.js file, add the highlighted lines to create a queue in Redis with BullMQ:

const { Queue } = require("bullmq");

const redisOptions = { host: "localhost", port: 6379 };

const imageJobQueue = new Queue("imageJobQueue", {
  connection: redisOptions,
});

async function addJob(job) {
  await imageJobQueue.add(job.type, job);
}

You start by extracting the Queue class from BullMQ, which is used to create a queue in Redis. You then set the redisOptions variable to an object with properties that the Queue class instance will use to establish a connection with Redis. You set the host property value to localhost because Redis is running on your local machine.

Next, you set the imageJobQueue variable to an instance of the Queue class, taking the queue’s name as its first argument and an object as a second argument. The object has a connection property with the value set to an object in the redisOptions variable. After instantiating the Queue class, a queue called imageJobQueue will be created in Redis.

Finally, you define the addJob() function that you will use to add a job in the imageJobQueue. The function takes a parameter of job containing the information about the job ⁠(you will call the addJob() function with the data you want to save in a queue). In the function, you invoke the add() method of the imageJobQueue, taking the name of the job as the first argument and the job data as the second argument.

Add the highlighted code to call the addJob() function to add a job in the queue:

app.post("/upload", async function (req, res) {
  const { image } = req.files;

  if (!image) return res.sendStatus(400);

  await addJob({
    type: "processUploadedImages",
    image: {
      data: image.data.toString("base64"),
      name: image.name,
    },
  });

  res.redirect("/result");
});

Here, you call the addJob() function with an object that describes the job. The object has the type attribute with a value of the name of the job. The second property, image, is set to an object containing the image data the user has uploaded. Because the image data in image.data is in a buffer (binary form), you invoke JavaScript’s toString() method to convert it to a string that can be stored in Redis.

Warning: Since Redis is an in-memory database, avoid storing large amounts of data for jobs in the queue. If you have a large file that a job needs to process, save the file on the disk or the cloud, then save the link to the file as a string in the queue. When BullMQ executes the job, it will fetch the file from the link saved in Redis.

You have now defined the functionality to create a queue in Redis and add a job. You also defined the processUploadedImages() function to process uploaded images, which will be called by a worker.

The remaining task is to create a consumer (or worker) that will pull a job from the queue and execute it. You will set this up in the next step.

Step 4: Adding a Dashboard to Monitor BullMQ Queues

In this step, you will use the bull-board package to monitor the jobs in the Redis queue from a visual dashboard. This package will automatically create a user interface (UI) dashboard that displays and organizes the information about the BullMQ jobs that are stored in the Redis queue. Using your browser, you can monitor the jobs that are completed, are waiting, or have failed without opening the Redis CLI in the terminal.

Open the index.js file in your text editor:

nano index.js

Add the highlighted code to import bull-board:

const { createBullBoard } = require("@bull-board/api");
const { BullMQAdapter } = require("@bull-board/api/bullMQAdapter");
const { ExpressAdapter } = require("@bull-board/express");

In the preceding code, you import the createBullBoard() method from bull-board. You also import BullMQAdapter, which allows bull-board access to BullMQ queues, and ExpressAdapter, which provides functionality for Express to display the dashboard.

Next, add the highlighted code to connect bull-board with BullMQ:

const serverAdapter = new ExpressAdapter();
const bullBoard = createBullBoard({
  queues: [new BullMQAdapter(imageJobQueue)],
  serverAdapter: serverAdapter,
});
serverAdapter.setBasePath("/admin");

First, you set the serverAdapter to an instance of the ExpressAdapter. Next, you invoke createBullBoard() to initialize the dashboard with the BullMQ queue data. You pass the function an object argument with queues and serverAdapter properties. The first property, queues, accepts an array of the queues you defined with BullMQ, which is the imageJobQueue here. The second property, serverAdapter, contains an object that accepts an instance of the Express server adapter. After that, you set the /admin path to access the dashboard with the setBasePath() method.

Next, add the serverAdapter middleware for the /admin route:

app.use("/admin", serverAdapter.getRouter());

The complete index.js file will now match the following:

const path = require("path");
const fs = require("fs");
const express = require("express");
const bodyParser = require("body-parser");
const fileUpload = require("express-fileupload");
const { Queue } = require("bullmq");
const { createBullBoard } = require("@bull-board/api");
const { BullMQAdapter } = require("@bull-board/api/bullMQAdapter");
const { ExpressAdapter } = require("@bull-board/express");

const redisOptions = { host: "localhost", port: 6379 };

const imageJobQueue = new Queue("imageJobQueue", {
  connection: redisOptions,
});

async function addJob(job) {
  await imageJobQueue.add(job.type, job);
}

const serverAdapter = new ExpressAdapter();
const bullBoard = createBullBoard({
  queues: [new BullMQAdapter(imageJobQueue)],
  serverAdapter: serverAdapter,
});
serverAdapter.setBasePath("/admin");

const app = express();
app.set("view engine", "ejs");
app.use(bodyParser.json());
app.use(
  bodyParser.urlencoded({
    extended: true,
  })
);
app.use(fileUpload());
app.use(express.static("public"));
app.use("/admin", serverAdapter.getRouter());

app.get("/", function (req, res) {
  res.render("form");
});

app.get("/result", (req, res) => {
  const imgDirPath = path.join(__dirname, "./public/images");
  let imgFiles = fs.readdirSync(imgDirPath).map((image) => {
    return `images/${image}`;
  });
  res.render("result", { imgFiles });
});

app.post("/upload", async function (req, res) {
  const { image } = req.files;

  if (!image) return res.sendStatus(400);

  await addJob({
    type: "processUploadedImages",
    image: {
      data: Buffer.from(image.data).toString("base64"),
      name: image.name,
    },
  });

  res.redirect("/result");
});

app.listen(3000, function () {
  console.log("Server running on port 3000");
});

After you are done making changes, save and close your file.

Run the index.js file:

node index.js

Return to your browser and visit http://localhost:3000/admin. The dashboard will load and display the status of your queues, including jobs that are waiting, completed, or failed.

You can now use the bull-board dashboard to monitor queues and track jobs in real-time.

Conclusion

In this article, you offloaded a time-intensive task to a job queue using BullMQ. First, without using BullMQ, you created an app with a time-intensive task that has a slow request/response cycle. Then you used BullMQ to offload the time-intensive task and execute it asynchronously, which boosts the request/response cycle. After that, you used Bull-Board to create a dashboard to monitor BullMQ queues in Redis.

You can visit the BullMQ documentation to learn more about BullMQ features not covered in this tutorial, such as scheduling, prioritizing or retrying jobs, and configuring concurrency settings for workers. You can also visit the Bull-Board documentation to learn more about the dashboard features.