How to do Job Scheduling with Redis and Node JS

Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, geospatial indexes with radius queries and streams.

To be precise, it is an in-memory key-value database.

We’ll be using Redis along with a Node JS package called Kue to schedule background process for a simple node js application. Background process usually helps in sending an Email/ notification or running any background process. For instance, in a movie ticket booking application, you need to notify a customer via an email when a seat is booked and remind him/her 10 minutes before the movie time. This will be our use case.

In this article, we’ll write a minimal REST API server with express js. The prerequisite to this blog will be to know how Node JS works and make sure that Redis is installed and running.

Subscribe to Sudo vs Root

Our newsletter rolls out every 15 days. No fluff. Pure content.

Setting up the project

We’ll be using npm init to set up a Node JS project.

First things first, let’s create a directory for your project. I have assumed the project name as node-js-job-scheduling. Using mkdir node-js-job-scheduling , create your directory.

Run npm init until the process is finished. You can also add the required attributes if you like to. You had nothing, but you have this now.

Let’s call NPM for help

We’ll be installing some npm packages to set up a simple express server and test out an API endpoint. Let’s hope it responds to our call on the first try.

Install express & body-parser to set up a simple server that listens to a port.

npm install express body-parser --save

Just Listen

Create a index.js file in the root of the project and set up a simple application like this:

const express = require("express"),  
 app = express(),  
 bodyParser = require("body-parser");  
  
// support parsing of application/json type post data  
app.use(bodyParser.json());  
  
app.get("/test", (req, res) => {  
 return res.json({ response: "It Worked!" });  
});  
  
app.listen(8080, () => console.log(\`Hey there! I'm listening.\`));

Let’s use Postman to test our API endpoints. To start our server, use nodemon or a command like node index.js . It’s recommended to use nodemon as that’ll automatically restart the server after every change made to the code.

Let’s check if the endpoint responds to our call, open Postman try this endpoint - http://localhost:8080/test

Nodemailer for Mails

Let’s install nodemailer to our project with the help of NPM.

npm i nodemailer --save

Create a file named nodemailer.js in the root of the project and add the following code that’ll send an email to the recipient when invoked. Let’s assume that this method only takes one argument which is the recipient’s email.

const nodemailer = require("nodemailer");

let send = async args => {
 try {
   // Generate test SMTP service account from ethereal.email
   // Only needed if you don't have a real mail account for testing
   let testAccount = await nodemailer.createTestAccount();

   // create reusable transporter object using the default SMTP transport
   let transporter = nodemailer.createTransport({
     host: "smtp.ethereal.email",
     port: 587,
     secure: false, // true for 465, false for other ports
     auth: {
       user: testAccount.user, // generated ethereal user
       pass: testAccount.pass // generated ethereal password
     }
   });

   // send mail with defined transport object
   let info = await transporter.sendMail({
     from: '"Sathish 👻" <sathish@example.com>', // sender address
     to: args.email, // list of receivers
     subject: args.subject, // Subject line
     html: args.body // html body
   });

   console.log("Message sent: %s", info.messageId);
   // Message sent: <b658f8ca-6296-ccf4-8306-87d57a0b4321@example.com>

   // Preview only available when sending through an Ethereal account
   console.log(nodemailer.getTestMessageUrl(info));
   return nodemailer.getTestMessageUrl(info);
   // Preview URL: https://ethereal.email/message/WaQKMgKddxQDoou...
 } catch (err) {
   console.log(`Error: ${err}`);
 }
};

module.exports = {
 send
};

Let’s test out if the Mail part works. We have used an ethereal username and password which will not send out an actual email but yet mimics it. We’ll be getting a link on how the mail will look.

Create an endpoint that invokes the method to send out an email. Before that let’s require our method in index.js

const nodemailer = require('./nodemailer')
app.post("/send-email", async (req, res) => {
  let emailLink = await nodemailer.send(req.body.email);
  return res.json({
    response: `Preview URL: ${emailLink}`
  });
});

Test our endpoint in postman using POST http://localhost:8080/send-email with the following body,

{
	"email": "varun@example.com"
}

The response will be the E-mail preview link from ethereal -

{
    "response": "Preview URL: https://ethereal.email/message/XLKY9ubtaQzq51cXXLKY-eRxKgyyADQoAAAAAT4cARKy3EizuCBzNIafNlg"
}

Kue for Job Scheduling

Let’s install the required NPM package to schedule a background process. For that, we need to install kue which is a priority job queue backed by Redis, built for node.js.

npm i kue --save

We’ll be writing an endpoint to schedule the background process/job and a worker that’ll process it. For our requirement, it will be sending an email. Let’s assume that a user is booking a ticket for a movie. He/she should be notified twice i.e. When the booking is confirmed (at the time of booking) and 10 minutes before booking.

For refactoring, we’ll separate the job schedule as two different methods. One will be to schedule the job and the other will be to process the scheduled job.

We’ll create a kue.js file that contains the following code block,

var kue = require("kue");
var Queue = kue.createQueue();

let scheduleJob = data => {
  Queue.createJob(data.jobName, data.params)
    .attempts(3)
    .delay(data.time - Date.now()) // relative to now.
    .save();
};

module.exports = {
  scheduleJob
};

The above code will schedule a job which will assign a name and it’s params. Also the delay. We will have one-second delay for rolling out the booking confirmation email and milliseconds timestamp which will be 10 minutes before the start time of the movie.

Job Workers

We’ll create another file named worker.js which will process the scheduled job based on its names. So, in our case, it will be the email job which will roll out emails.

var kue = require("kue");
var Queue = kue.createQueue();
var nodemailer = require("./nodemailer");

Queue.process("sendEmail", async function(job, done) {
  let { data } = job;
  await nodemailer.send(data);
  done();
});

The above code block will make use of the nodemailer utility function we defined previously in this article and send out the email. Note that the arguments/ parameters we passed when scheduling the job will be as an attribute in the first argument of the queue process callback function with the key data. So, we get that with the help of object destructuring - let { data } = job; , and pass it to the nodemailer utility function. Require/Import the worker in index.js file which is our project’s entrypoint.

API Endpoint

We’ll now write an API endpoint /book-ticket that will schedule the job for us in index.js. It’ll schedule two jobs for us. One is to send out the booking confirmation E-mail that has a delay of say, one-second and the second one is to send an email 10 minutes before the booking.

const express = require("express"),
 app = express(),
 bodyParser = require("body-parser");
const kue = require("./kue");
require("./worker");

// support parsing of application/json type post data
app.use(bodyParser.json());

app.post("/book-ticket", async (req, res) => {
 let args = {
   jobName: "sendEmail",
   time: 1000,
   params: {
     email: req.body.email,
     subject: "Booking Confirmed",
     body: "Your booking is confirmed!!"
   }
 };
 kue.scheduleJob(args);

 // Schedule Job to send email 10 minutes before the movie
 args = {
   jobName: "sendEmail",
   time: (req.body.start_time - 10 * 60) * 1000, // (Start time - 10 minutes) in millieconds
   params: {
     email: req.body.email,
     subject: "Movie starts in 10 minutes",
     body:
       "Your movie will start in 10 minutes. Hurry up and grab your snacks."
   }
 };
 kue.scheduleJob(args);

 // Return a response
 return res.status(200).json({ response: "Booking Successful!" });
});

app.listen(8080, () => console.log(`Hey there! I'm listening.`));

The above endpoint makes use of the utility function we wrote to schedule a job. It has a job name, the time (delay) and the email args which is required to send out the email.

The first job is about the confirmation email after a ticket is booked with a delay of one second which is 1000 milliseconds. The second job is about the notification email that should be sent 10 minutes before the start time of the movie. So the delay time is (req.body.start_time - 10 * 60) * 1000 where the start time is a unix timestamp from the request body. The request body will also contain the email of the user to whom it should be sent to.

The request body will be as the following,

{
	"email": "sathish@skcript.com",
	"start_time": 1555974600
}

Let’s test it again with Postman

The Ethereal preview link will be printed in the console as follows,

Message sent: <10360fc7-22bb-017e-89b5-a4e72ea78a43@example.com>
https://ethereal.email/message/XL7wFeJAYA0Dt0R1XL7wGIdA2wzOT3MhAAAAAbyCsQRrhWTVaptU0njFIO0
Message sent: <f08c907d-6009-41c9-9b59-869a6bf571da@example.com>
https://ethereal.email/message/XL7wFeJAYA0Dt0R1XL7wGg7lLAzJdYNkAAAAAkhU1sH5NPPfG5.x8kRnio8

The confirmation E-mail

The Notification E-mail

Folder Structure

Our journey ends here and I hope you found this article helpful. You can also find the project on Github or follow me on Twitter for some funny retweets.

Comments and Discussions

Skcript https://www.skcript.com/svr/how-to-do-job-scheduling-with-redis-and-node-js/ https://www.skcript.com/svrmedia/heroes/redis-and-job-scheduling-in-node-js@1.5x.png