Let’s build a Node.js API that uses Bull to handle priority Jobs
Why Bull?
There are a wide variety of priority job queues for Node js.
“A Queue is a linear structure which follows a particular order in which the operations are performed. The order is First In First Out (FIFO). A good example of a queue is any queue of consumers for a resource where the consumer that came first is served first”. – GeeksforGeeks
I have played around with Kue in the past but the package hasn’t been updated in about 4 years, and they now recommend that you try bull. When building an API, the aim is always to return a response in the fastest possible time. But there are times when we want to run some jobs based on the information we have in the logic of our code at a later time. This is where Bull comes in.
Bull is a feature-rich priority job queue for Node.js backed by Redis. It enables us to delegate that process/job to Redis to run in the background while returning a response immediately. Bull follows the First in First out format, but we can set a priority for our jobs in the configuration, and the job with a higher priority would take precedence over the one with a lower one.
Brief Overview of What We’ll Be Building
In this article, we’ll be building an API that registers users and send them an email alongside a text message on successful registration. We’ll be using Kue Priority Jobs to send the Welcome email and text message with the help of Sendgrid and Twilio Messaging API.
Prerequisites
Setting up our Server
We’ll be building this demo using ExpressJs, a minimal and flexible Node.js web application framework that provides a robust set of features for web and mobile applications. It’s important to have Node already installed on your local machine. To verify that you have node installed, open up your terminal and run the following command:
node --vIf you don’t have Node installed, you can download it here.
Next, we’ll need to create a folder to house our server-side code, and we can do that by running the command in our terminal below:
mkdir <Name of folder>Then, we change the directory into our new folder, open it in our code editor and run the init command to initialize our node app and create a package.json file for us.
npm initAfter initializing our app and creating our package.json file, we’ll install the following packages to get started:
npm i express redis bull mongoose dotenv twilio @sendgrid/mail mailgenAfter installing our dependencies, we’ll need to write the code to start up our server in our index.js file as seen below:
require("dotenv").config();
const express = require("express");
const app = express();
const port = process.env.PORT || 4000;
app.listen(port, () => console.log(`App listening on port ${port}`));Setting up File Structure
Now, we are going to set up the file structure for our code.
In order for this to work, you need to have a local installation of MongoDB and Redis. You can download a local installation of MongoDB here, and that of Redis here. Alternatively, you can set up a free tier of MongoDB on atlas using this link, and use their connection link. We are also going to set up our database config, which will include configs for connecting to MongoDB, as seen below:
require("dotenv").config();
const mongoose = require("mongoose");
let mongoUrl = null;
const mongoConnection = () => {
if (process.env.NODE_ENV === "test") {
mongoUrl = process.env.TEST_DB;
} else {
mongoUrl = process.env.DATA_DB;
}
return mongoose.connect(mongoUrl, {
useNewUrlParser: true,
useUnifiedTopology: true,
useCreateIndex: true,
useFindAndModify: false,
});
};
module.exports = mongoConnection;The code in the file above holds the logic to connect to our MongoDB database.
Next, we are going to flesh out our user schema as seen below:
const mongoose = require("mongoose");
const userSchema = new mongoose.Schema(
{
firstName: {
type: String,
required: [true, "User must have a first Name"],
trim: true,
lowercase: true,
},
lastName: {
type: String,
required: [true, "User must have a last Name"],
trim: true,
lowercase: true,
},
sex: {
type: String,
required: [true, "Sex is required"],
trim: true,
lowercase: true,
},
email: {
type: String,
unique: true,
required: [true, "User must have an email"],
trim: true,
lowercase: true,
},
dateOfBirth: {
type: Date,
required: [true, "User must have a Date of Birth"],
},
phoneNumber: {
type: String,
required: [true, "User must have a Phone Number"],
},
},
{ timestamps: true }
);
module.exports = mongoose.model("User", userSchema);Registering users and using Kue Priority Jobs to send the Welcome email and text message
We will now go-ahead to create a register file within our controllers to handle the logic that creates our users. We are also going to be sending emails and text messages to our users on registration before returning a response using Bull.
To do this, follow these steps:
- First, we are going to create some helper functions to help us send welcome email and text messages to our newly registered users
To send welcome emails, you can refer to the helper function below:
require("dotenv").config();
const Mailgen = require("mailgen");
const sgMail = require("@sendgrid/mail");
sgMail.setApiKey(process.env.SENDGRID_API_KEY);
function welcomeEmail(job, done) {
const mailGenerator = new Mailgen({
theme: "default",
product: {
name: "Aeeiee",
link: `https://aeeiee.com`,
},
});
const mail = {
body: {
name: `${job.firstName} ${job.lastName}`,
intro: "Welcome to Aeeiee! We're very excited to have you on board.",
action: {
instructions: "To get started with Aeeiee, please click here:",
button: {
color: "#22BC66",
text: "Checkout our Website",
link: `https://aeeiee.com`,
},
},
outro:
"Need help, or have questions? Just reply to this email, we'd love to help.",
},
};
const emailBody = mailGenerator.generate(mail);
const emailText = mailGenerator.generatePlaintext(mail);
const mailOption = {
to: job.email,
from: `${process.env.SENDERS_EMAIL}`,
subject: "Welcome to Aeeiee",
html: emailBody,
text: emailText,
};
done();
return sgMail.send(mailOption);
}
module.exports = { welcomeEmail };Likewise, to send text messages using Twilio, you can use the helper function below. If you don’t have Twilio set up, you can follow this link to find out how to set up Twilio and create your API credentials.
const { errorHelper } = require("./response");
require("dotenv").config();
const accountSid = process.env.TWILIO_ACCOUNT_SID;
const authToken = process.env.TWILIO_AUTH_TOKEN;
const client = require("twilio")(accountSid, authToken);
async function sendText(job, done) {
try {
await client.messages.create({
body: `Welcome to Aeeiee, ${job.firstName}`,
from: `${process.env.MESSAGING}`,
to: `${job.phoneNumber}`,
});
return done();
} catch (error) {
done();
return errorHelper(res, 500, error);
}
}
module.exports = {
sendText,
};- Next, we are going to create a Bull config file to handle the logic of our Job Queue.
const Queue = require("bull");
const queue = new Queue("Jobs");
const { welcomeEmail } = require("../helpers/welcomeEmail");
const { sendText } = require("../helpers/twilio");
const createJob = (options, data) => {
const opts = { priority: 0, attempts: 5};
queue.add(options, data, {
attempts: opts.attempts,
backoff: {
type: "exponential",
delay: 2000,
},
removeOnComplete: true,
removeOnFail: true,
});
};
queue.process("Welcome email", (job, done) => welcomeEmail(job.data, done));
queue.process("Send Text", (job, done) => sendText(job.data, done));
module.exports = { createJob };The above code shows some of the configurations we are setting for the instance of our Bull. Our job is going to re-attempt to run 5 times in the event of a failure. We are also setting the job to be removed from the queue on completion or on failure. And finally, we have our backoff settings to help with third-party rate limiting.
- Then we create the register controller that helps us register our users
const { successResponse, errorHelper } = require("../helpers/response");
const models = require("../../database/models");
const { createJob } = require("../Jobs/bullConfig");
async function createUser(req, res) {
try {
const user = await models.User.create(req.body);
createJob("Welcome email", user);
createJob("Send Text", user);
return successResponse(res, 201, "User Registered Successfully", user);
} catch (error) {
return errorHelper(res, 500, error);
}
}
module.exports = { createUser };Then we expose this controller by creating a route that exposes it. We are going to do this by creating an endpoint in our route folder as shown below:
const express = require("express");
const bodyParser = require("body-parser");
const { createUser } = require("../controllers/register");
const app = express();
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: false }));
app.post("/register", createUser);
app.get("/", (req, res) => {
res.send("Hello World");
});
module.exports = app;Our register endpoint should be available at http://localhost:4000/register. We can now call our endpoint using Postman with the necessary data, and get back a response as shown below.
We will also receive an email and text message as shown below which is proof that our jobs are running.
This is a very basic example of using Bull to create and schedule Jobs. There are more options in bull such as Event listeners, Rate limiters, Job types, using Cron expressions, etc. Visit the official documentation for more information. The full code can be found here
