Redis · Chapter 19 of 26
Locks, one worker at a time
By the end of this chapter you can take a lock in Redis with one SET, give it a lease so a crashed worker cannot hold it for ever, release only your own lock, wait for a busy lock with backoff, stop a slow worker doing damage with a fencing token, and say what Redlock is for. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Redis in production
- 4 tasks
- Redis
- In the all labs plan
The first 2 chapters of Redis are free. This one comes with the all labs plan.
Why you need this
When match 2001 ends, Venkat's payout job pays the winners of every contest: ₹11,000 in the Mega Contest alone. GullyXI runs the job on two workers, so it still runs if one machine is down. Usually that is fine. Last week both started it at the same moment for contest 74, and the winners were paid twice. Getting that money back from fans took a week of apologetic emails.
What the two workers need is a lock: a sign on the door that says "someone is doing this", which only one of them can hang up at a time, which the other can see, and which comes down by itself if the worker holding it dies. A row in the main database could do it, slowly, and with nothing to remove it when a worker crashes. Every app server already talks to Redis, and Redis runs one command at a time, which is exactly what "only one of them" needs.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
acquire.js: take a lock
Export acquireLock(redis, name, ttlMs). Take the lock lock:<name> (lock:payout:71 for the name payout:71) with one command that sets it only if it does not exist and gives it a lease: it expires by itself after ttlMs milliseconds (the Check reads a PTTL above ttlMs minus 1000 and at most ttlMs).
The value is a new random token for every acquire, at least 16 characters (crypto's randomUUID gives 36). Return the token when you got the lock, and null when it was taken (the key stays as it was). Five acquires of one lock at the same moment: exactly one gets a token.
- Task 2 · interview question
Interview: release.js, release only your own lock
releaseLock(redis, name, token) releases with a plain DEL of lock:<name>. Contest 74 was paid twice: worker A's 10 s lease ran out while the bank was slow, worker B took the lock, A finished and its DEL deleted B's lock, and worker C took it and paid again.
Fix it: delete lock:<name> only if it still holds token, and return true when you deleted it, false otherwise (another worker's lock stays exactly as it is, and a lock that is gone gives false). Compare and delete in one script (chapter 18), key in KEYS and token in ARGV; the Check records the commands your function sends and accepts only scripts.
Farhan will ask: why is releasing a lock with a plain DEL wrong?
- Task 3
with-lock.js: a withLock helper
Export withLock(redis, name, ttlMs, fn, { retries = 5, delayMs = 50 } = {}), using your acquireLock and releaseLock (imported from Tasks 1 and 2). Try to acquire lock:<name> with ttlMs. While it is busy, wait and try again, up to retries more times, waiting delayMs the first time and twice as long each time after (50, 100, 200, 400, 800 ms by default). If it is still busy, throw new Error('lock busy: <name>') (for example 'lock busy: job:c') without calling fn and without touching the other worker's lock.
With the lock, return await fn(), and release the lock whether fn returned or threw; if fn threw, the same error goes on to the caller.
The Check gives up after retries 2 and delayMs 30 (it must wait at least 80 ms in all), gets a lock that frees after 120 ms with the defaults, and runs two withLock calls on one lock at once: both finish, never at the same time.
- Task 4
payout.js: the payout runs once
Export runPayout(redis, contestId, pay). Use your withLock on the name payout:<contestId> (so the key is lock:payout:<contestId>), with ttlMs 10000 and the default retries. Inside the lock: if payout:<contestId>:done exists, return 'skipped'; otherwise take a fencing token by adding 1 to payout:<contestId>:fence (it starts at 1), await pay(token), then set payout:<contestId>:done and return 'paid'.
If withLock gives up ('lock busy: payout:<contestId>'), return 'skipped'. If pay throws, the error goes on to the caller and done is not set, so the next run pays, with fencing token 2.
The Check starts two workers for contest 71 at once, with a pay that takes 100 ms: one returns 'paid', the other 'skipped', pay runs once with token 1, and a third worker later returns 'skipped'.
What you will learn
- SET with NX and PX: take a lock in one command, with a lease.
- A random token: a lock that knows who holds it.
- Releasing your own lock only: chapter 18's compare-and-delete.
- Retrying with backoff: waiting for a busy lock without hammering Redis.
- Fencing tokens: INCR numbers that stop a worker whose lease ran out.
- Redlock, in a paragraph.
More in Redis in production
Build "Locks, one worker at a time" yourself
GullyXI, a Pune fantasy cricket app, falls over on match night. You make it fast with Redis. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Redis are free. This one comes with the all labs plan.