Build an AI gateway (TypeScript) · Chapter 17 of 21
Traces, percentiles and logs
By the end of this chapter you can follow one request through Darwaza step by step, tie every provider call to the request that made it, work out which model is slow from Darwaza's own records, and write a log line that helps at 2 am without holding a key or a customer's phone number. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Run it in production
- 4 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Build an AI gateway (TypeScript) are free. This one comes with the all labs plan.
Why you need this
Darwaza now routes, retries, falls back, caches, limits and streams. Each layer made Jawab's numbers better, and each one made a single request harder to follow. At 14:10, during a bad minute at Kestrel and Nimbus's outage, a Kayal Pickles FAQ was turned away by kestrel-swift twice, waited 800 ms on nimbus-mini, gave up, and was answered by tara-flash. The customer saw a slow answer. Nobody else saw anything.
Joseph, who is on call, asks three questions every time something feels slow: which request, which step, and which provider. "When a customer says the reply took a second, I want to see the request: every call it made, in order, how long each took and how it ended. When Kestrel asks me which call failed, I want to give them something they can find. And I want a log I can search at 2 am that never shows me a key or a customer's Aadhaar number."
Darwaza so far answers none of that. printReport says how the day went on average; it cannot say what happened to one request. The providers log what they received, but they cannot tell which of Darwaza's requests a call belonged to. And nothing is written down at all.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
trace.ts: a trace, and its spans
Write and export class Trace: traceId (32 lowercase hex characters from 16 random bytes), spans (every span, in the order they start) and root (the first span, started by the constructor with its name and attrs).
start(name, attrs = {}, parent?) returns a new span: { traceId, spanId (16 hex characters from 8 random bytes), parentId (the parent's spanId, or the root's when no parent is given; the root has none), name, start: performance.now(), attrs } and pushes it into spans. end(span, attrs = {}) sets end to performance.now() and adds attrs to the span's. traceparent(span) returns
00-${traceId}-${span.spanId}-01. - Task 2
traced.ts: a trace for every request
Add tracing to the given request loop in traced.ts. Export const traces: Trace[]. At the top of handle, start new Trace('gateway', { team, feature }) and push it into traces.
For every provider call, start a span named 'attempt' with 'gen_ai.provider.name' and 'gen_ai.request.model' (the target), pass headers: { traceparent: trace.traceparent(span) } to call, and end the span with 'http.response.status_code' (200 or the error's status), plus 'gen_ai.usage.input_tokens' and 'gen_ai.usage.output_tokens' when it answered, or 'error.type' (the error's kind) when not. Before every return, end the root span with 'http.response.status_code' and, when answered, 'gen_ai.response.model'.
- Task 3
percentiles.ts: p50, p95 and p99 per model
Write and export latencyByModel(traces): Record<string, { calls, p50, p95, p99 }>, from the spans named 'attempt' that have an end and ended with 'http.response.status_code' 200, grouped by 'gen_ai.request.model'. Each span's time is Math.round(end - start); the percentiles are by nearest rank.
Under if (import.meta.main), replay 09:00 to 12:00 through handle from ./traced.ts, printReport it, then print one line per model, most calls first: the model (provider/model), calls N, p50 N ms, p95 N ms, p99 N ms.
- Task 4 · interview question
Interview: log.ts, a log line for 2 am
Write and export redact(text): every secret replaced by [secret] and the rest kept: Darwaza's three keys wherever they appear, the token after Bearer, the value after ?key= or &key=, and the value of an x-api-key or x-tara-api-key header (x-api-key: v or "x-api-key":"v").
Write and export logLine(trace, req, res): JSON.stringify of { trace, team, feature, status, model (when answered) or error (when not), ms (the root span's end - start, rounded), attempts (the number of attempt spans) }, run through redact. Never the messages. Under if (import.meta.main), send day({ tags: ['pii'] })[0].request through handle from ./traced.ts and print its log line.
What you will learn
- Traces and spans: one request as a tree of timed steps.
- Trace ids and span ids, and the W3C traceparent header that carries them to the providers.
- OpenTelemetry's names for what an AI call records (gen_ai.*).
- Percentiles per model, from Darwaza's own spans.
- A log line: what goes in, what never goes in, and redaction.
More in Run it in production
Build "Traces, percentiles and logs" yourself
Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build an AI gateway (TypeScript) are free. This one comes with the all labs plan.