Redis · Chapter 15 of 26
Streams, the ball-by-ball feed
By the end of this chapter you can keep GullyXI's ball-by-ball feed in a Redis stream: add balls with ids Redis makes, keep the stream short, read a range or everything after an id, wait for the next ball, and share the balls between workers with a consumer group that knows which balls are done and which are stuck. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Redis from Node.js
- 4 tasks
- Redis
- In the all labs plan
The first 2 chapters of Redis are free. This one comes with the all labs plan.
Why you need this
Tonight Mumbai bat first in match 2001, and every one of their 120 balls has to reach three places. Rhea wants each fan's phone to show the ball within a second, and a phone that lost its signal in the 6th over must catch up on what it missed, not jump to the latest ball. Venkat needs the fantasy points service to score every ball exactly once: a ball scored twice gives fans points they did not earn, and a ball missed costs someone a prize. And Gurpreet knows a points worker will crash mid-over at some point; its balls must not vanish with it.
Chapter 14's list queue gets you part of the way. A popped job is gone, which is right for a job, but here two services need every ball: with lists you would push each ball twice, into two lists, and a phone that reconnects cannot ask for "everything after ball 39", because a list has positions, not names that stay put. Redis has a type made for this: the stream, an append-only log where every entry keeps a name forever.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
publish-innings.js: the innings as a stream
Export publishInnings(redis, matchId, balls). Add each ball, in order, as one entry of the stream match:<matchId>:balls, with an id Redis makes from its clock (the * id; the Check checks the ids are close to the time now). The entry's fields are every field of the ball, in the ball's order, each value as text: seq, over, batter, bowler, runs, wicket, total, wickets for balls.json, so false becomes "false" and 6 becomes "6".
Trim on every add with the approximate MAXLEN, to about 1000 entries: after 1300 balls the Check accepts a length from 1000 to 1100. Return the ids Redis gave, in order.
The Check publishes the 120 balls of balls.json for match 2001 and reads them back with XRANGE.
- Task 2
next-balls.js: read from an id onwards
Export nextBalls(redis, matchId, lastId, blockMs = 1000). Read the balls of match:<matchId>:balls that come after lastId ('0' means from the start), at most 20 of them, waiting up to blockMs milliseconds for one when there is none yet.
Return them in order as objects: { id, ...fields }, the entry's id and then every field with its text value, for example { id: '1792709400001-0', seq: '1', over: '0.1', ... }. When no ball comes within blockMs, return [] (Redis answers nil, which ioredis gives as null).
The Check reads after the first of three balls (two come back), from '0' on 28 balls (20 come back), waits 300 ms with nothing new (it must really wait), and adds a ball while you wait (it comes back at once).
- Task 3
points-workers.js: two workers share one group
Export createGroup(redis, matchId): make the consumer group points on match:<matchId>:balls, reading from the very first entry (id 0, so balls added before the group still count), and make the stream too when it does not exist yet. Return true when it made the group, and false when the group was already there (Redis answers BUSYGROUP): never throw for that.
Export workBalls(redis, matchId, consumer, handle, count = 10): read, as that consumer of the points group, up to count balls no one in the group has been given yet. For each ball, in order, await handle({ id, ...fields }) (fields as text, as in Task 2), then acknowledge the ball. If handle throws, leave that ball unacknowledged (it stays pending) and carry on with the next. Return the ids you acknowledged, in order, and [] when no ball came.
The Check adds 15 balls before making the group: w1 with count 10 gets the first ten, w2 the other five, and nothing is left pending.
- Task 4 · interview question
Interview: stuck-worker.js, the worker that died mid-over
Worker w1 read three balls from the points group and died before acknowledging them, so they stay pending for w1 and no normal read will give them to anyone else.
Export rescueBalls(redis, matchId, consumer, minIdleMs, handle): take over, as consumer, the balls in the points group of match:<matchId>:balls that have been pending for at least minIdleMs (up to 100, from the start of the pending list). For each, in order, await handle({ id, ...fields }) and acknowledge it. Return the ids handled, and [] when none was idle long enough.
The Check lets w1 take three balls and w3 take and acknowledge two. With minIdleMs 60000 straight away nothing comes back; after 150 ms, with minIdleMs 100, w1's three balls come back and nothing is left pending.
Farhan will ask: chapter 14 built a queue on a list. For the ball feed, a list queue or a stream, and why?
What you will learn
- XADD and ids: add an entry, with an id Redis makes from its clock.
- XRANGE, XREVRANGE and XLEN: read a range of entries, newest first, and count them.
- MAXLEN ~: keep a stream near a length, cheaply.
- XREAD with BLOCK: read after an id, and wait for the next entry.
- Consumer groups: XGROUP CREATE, XREADGROUP with >, and XACK.
- XPENDING and XAUTOCLAIM: see the entries a worker never finished, and take them over.
More in Redis from Node.js
Build "Streams, the ball-by-ball feed" yourself
GullyXI, a Pune fantasy cricket app, falls over on match night. You make it fast with Redis. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Redis are free. This one comes with the all labs plan.