Go · Chapter 25 of 26
Capstone: the ingest service
By the end of this chapter you have built BijliGo's new ingest service end to end, from the pieces of the whole lab: an HTTP handler that never waits, a buffered queue, a pool of writers that store meter readings in Postgres in batches, a status cache in front of the slow central system, chapter 22's middleware around the API, and a shutdown that loses nothing. A replay of ST001's 1,263 readings of Wednesday drives it, and the Check compares the database with the data. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Services
- 5 tasks
- Go
- In the all labs plan
The first 2 chapters of Go are free. This one comes with the all labs plan.
Why you need this
On Wednesday 14 October the old ingest service took every message one at a time. At the evening peak its line grew past 30 seconds, chargers that heard nothing back marked themselves offline, Tenzin was paged at 7:12 pm, and 109 sessions failed. Nandini has set the date for the switch to the new service, and Joseph has one rule for it: "It is the same Go you have written for 24 chapters. Nothing new, only all of it at once."
What the service must do, in Tenzin's words: take a reading in well under a second and never make a charger wait; never lose a reading it said yes to; tell a partner app a charger's status without asking the central system 40 ms every time; log every request; let in only apps with a key; and stop for a deploy without dropping what is queued.
What you build: 5 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
readings.go: take a reading, never wait
In readings.go, write ReadingsHandler(queue chan<- bijli.Reading) http.Handler. Each request's body is one bijli.Reading as JSON, for example {"session":"S900710","charger":"C1002","at":"2026-10-14T06:42:00+05:30","meter_wh":410,"power_w":49200}.
202 (Accepted): the reading is put on queue. 400 (Bad Request), with nothing queued: the body is not JSON, has a field a Reading does not have (like volts), or has no session, no charger, no at, or a meter_wh below 0. 503 (Service Unavailable): queue is full; answer at once, never wait for room.
The Check uses a queue with room for 2: the reading must arrive on the queue as it was sent, and the third good reading while two are queued gets 503.
- Task 2
writer.go: write the readings to Postgres, in batches
In writer.go, write WriteReadings(ctx context.Context, pool *pgxpool.Pool, queue <-chan bijli.Reading, workers, batchSize int) error. It starts workers goroutines that take readings from queue and store them in the readings table (session_id, charger_id, at, meter_wh, power_w). Each worker collects up to batchSize readings and writes them in one transaction; a part batch is written within about 50ms even when no more readings come; a reading already stored (the primary key is session_id and at) is skipped, not an error.
When queue is closed, the workers write what is left and WriteReadings returns nil once all are done. When ctx ends it returns an error that is ctx's error (errors.Is finds context.Canceled). When a write fails it returns that error. Chapter 20's errgroup (golang.org/x/sync/errgroup) runs the workers well.
The Check empties the readings table, sends 3 readings and waits up to 1.5s for them to be stored with the queue still open, closes the queue (nil within 2s); then sends all 1263 of ST001's readings with 10 of them twice, with 4 workers and batches of 100, and compares the table with the data; then cancels ctx with the queue open (it must return within 2s).
- Task 3
status.go: a status cache in front of the central system
In status.go, write the type StatusCache (add the fields you need), NewStatusCache(central bijli.Central, ttl time.Duration, now func() time.Time) StatusCache, and the method Get(ctx context.Context, id string) (bijli.Status, error). Get returns the charger's status from the cache when it was fetched less than ttl ago (by now()), and otherwise asks central.ChargerStatus(ctx, id) and keeps the answer with the time it was fetched.
Errors come back as they are (bijli.ErrUnknownCharger, bijli.ErrChargerDown, ctx's error) and are not cached. The cache is safe for many goroutines.
The Check uses a clock it moves by hand and a ttl of 30s: Gets of C1042 at 0s, 0s and 29s make 1 call to the central system, a Get at 30s makes the second; 50 goroutines then Get C1042 at once with no call; two Gets of C9999 make two calls; C1099 is down; a cancelled ctx returns context.Canceled. All under the race detector.
- Task 4
server.go: the service's API, with its middleware
In server.go, write NewServer(logger slog.Logger, apiKey string, readings http.Handler, statuses StatusCache) http.Handler. Routes: POST /readings goes to readings. GET /chargers/{id}/status answers 200 with Content-Type application/json and {"charger":"C1042","station":"ST007","state":"available","power_kw":30} from statuses.Get; 404 for bijli.ErrUnknownCharger, 502 (Bad Gateway) for bijli.ErrChargerDown, 500 for any other error.
Every request needs the header X-API-Key equal to apiKey, else 401 with http.Error and "missing or wrong API key", before any route runs. Every request is logged with logger at Info with the message "request" and the attributes method, path and status (the status the client got), refused ones too. A panic becomes a 500 and its request is still logged with status 500. Copy chapter 22's middleware into this package.
The Check passes its own readings handler (one that panics for the body panic) and a StatusCache over a central system where C1099 is down.
- Task 5
service.go: the whole service, and a clean shutdown
In service.go, write the type Service (add the fields you need) and Start(ctx context.Context, pool pgxpool.Pool, central bijli.Central, logger slog.Logger, apiKey string) Service. Start makes a queue with room for 1000 readings, runs WriteReadings on it with 4 workers and batches of 100 under ctx in its own goroutine, makes a StatusCache with a ttl of 30s and time.Now, and builds NewServer with your ReadingsHandler on the queue.
Handler() http.Handler returns the API. Shutdown(ctx context.Context) error stops taking readings (POST /readings answers 503 from then on), lets the writers store everything queued, and returns their error (nil), or ctx's error if ctx ends first. A request that arrives while Shutdown closes the queue must get 503, never a send on a closed channel.
The Check empties the readings table, GETs /chargers/C1051/status (200), replays ST001's 1263 readings through Handler() from 8 goroutines with the key (a 503 is tried again), calls Shutdown while 4 goroutines are still posting (every answer must be 202 or 503), posts once more after Shutdown (503), and compares the readings table with the data. Tasks 1 to 4 must pass first.
What you will learn
- Handlers and JSON (chapters 12 and 21): POST /readings decodes a bijli.Reading.
- Buffered channels and select (16 and 17): a queue the handler never waits on.
- Worker pools, errgroup and tickers (17, 19 and 20): writers that batch and stop with ctx.
- pgx and transactions (23): one transaction per batch, a reading stored once.
- Locks and the race detector (18): a cache, and a shutdown with no data race.
- Middleware (22): the key check, the request log and the panic guard, in the right order.
- Time passed in (14): a cache tested without waiting 30 seconds.
More in Services
Build "Capstone: the ingest service" yourself
BijliGo, a Bengaluru EV charging network, drops chargers at the evening peak. You rewrite its backend in Go. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Go are free. This one comes with the all labs plan.