Redis · Chapter 14 of 26

Queues

By the end of this chapter you can run a job queue on a Redis list from Node.js: app servers push jobs with RPUSH, workers wait for them with BLPOP on a connection of their own, a reliable worker keeps each job in a processing list with BLMOVE until it is done, a dead worker's jobs go back to the queue, and a job that keeps failing ends in a dead-letter list. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Redis from Node.js
  • 4 tasks
  • Redis
  • In the all labs plan

The first 2 chapters of Redis are free. This one comes with the all labs plan.

Why you need this

After every over, the points change, and every fantasy team in every contest has to be scored again: 449 teams in match 2001's five contests, each a little work. Done inside the request that receives the ball, it would hold that request for as long as all the scoring takes, and one slow team would hold up the rest. So GullyXI splits it: the app server puts one job a team on a queue and moves on, and a few workers take jobs off the queue and do them, one at a time, oldest first.

The queue must be shared by every app server and every worker, which is Redis again. Chapter 5 taught that a list with RPUSH at one end and LPOP at the other is a queue. This chapter makes it work for real: workers that wait without spinning, and a queue that does not lose a job when a worker dies in the middle of it. Last week a worker was restarted during a deploy, and Venkat found nineteen teams that were never scored.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    queue.js: a queue of points to calculate

    In queue.js, write enqueue(redis, job): add the job (an object such as { contest: 71, fan: 'u1056', over: '5.6' }) to the end of the list queue:points as JSON text, and return the queue's length after adding it (1, 2, 3 for three jobs).

    Then write workOne(redis, handler), which a worker calls on its own connection: take the oldest job from the front of queue:points with BLPOP, waiting up to 1 second for one, then await handler(job) with the job as an object, and return the job. If no job comes within 1 second, return null without calling the handler. A job pushed while workOne is waiting must be worked at once.

  2. Task 2 · interview question

    Interview: lost-job.js, the job a crash lost

    In lost-job.js, workReliably(redis, workerId, handler) takes a job off queue:points with BLPOP, so a worker that crashes mid-job loses it: it is in no list at all. Fix it: in one command (BLMOVE), take the oldest job from the front of queue:points and put it at the end of queue:points:processing:<workerId> (queue:points:processing:w1), waiting up to 1 second. Then await handler(job) with the job as an object. When the handler is done, remove that job from the processing list and return the job.

    If the handler throws, leave the job in the processing list exactly as it was taken, and let the error go up (workReliably rejects). If no job comes within 1 second, return null. Other workers' processing lists stay as they are.

    Farhan will ask: what happens to a job when a worker dies in the middle of it, with the old code and with yours?

  3. Task 3

    recover.js: put back a crashed worker's jobs

    In recover.js, write recoverWorker(redis, workerId): move every job in queue:points:processing:<workerId> back to the front of queue:points, so they run next. The worker's oldest job (the one it took first) must end up first, then its others in order, then the jobs that were already waiting. Move each job with one command, so a job is always in one list or the other.

    Return how many jobs it put back (0 for a worker with no jobs). Other workers' processing lists stay as they are.

  4. Task 4

    retries.js: three tries, then the dead-letter list

    In retries.js, write workWithRetries(redis, workerId, handler). Take a job as in Task 2: from queue:points into queue:points:processing:<workerId>, waiting up to 1 second (null if none comes). If the handler works, remove the job from the processing list and return 'done'.

    If the handler throws, remove the job from the processing list (by the exact text that was taken) and add 1 to the job's attempts field (a missing field means 0). With attempts below 3, push the job with its new attempts to the end of queue:points and return 'retry'. At 3, push it to queue:points:dead with attempts and error (the error's message), and return 'dead'. Do the remove and the push in one MULTI. workWithRetries never throws for a failed job.

    A job that always fails gives 'retry', 'retry', 'dead', and ends as one JSON in queue:points:dead such as { contest: 71, fan: 'u9999', over: '7.2', attempts: 3, error: 'no such fan' }.

What you will learn

  • RPUSH and BLPOP from ioredis: adding jobs, and waiting for one with a timeout.
  • A worker's own connection: why a blocking command needs redis.duplicate().
  • JSON jobs: a job is an object, so it travels as JSON text.
  • BLMOVE and a processing list: a job is never in neither list.
  • LREM: removing a finished job by its exact text.
  • Recovery, retries and a dead-letter list: what to do with a dead worker's jobs, and with a job that always fails.

More in Redis from Node.js

Build "Queues" yourself

GullyXI, a Pune fantasy cricket app, falls over on match night. You make it fast with Redis. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Redis are free. This one comes with the all labs plan.