Build an AI gateway (TypeScript) · Chapter 19 of 21

Load testing the gateway

By the end of this chapter you can load-test Darwaza the way Jawab's app will use it, over HTTP; tell an honest load test from one that hides the worst minutes; find how much traffic Darwaza can take before the SLO breaks; and check the SLO for each brand, not only for Jawab as a whole. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Run it in production
  • 4 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Build an AI gateway (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Every chapter so far measured Darwaza by calling its handle function in the same program as the replay. Jawab's app will not do that: it will send HTTP requests to Darwaza running as a service. And Tuesday is only one day. Anjali has signed two more brands for next month, and Payyoli is planning a bigger sale. Her question is simple: "How much more can Darwaza take before customers notice?"

Joseph has a second question. A friend's team load-tested their gateway with a popular tool, saw p95 under 600 ms, and went live; the first outage gave customers six-second waits the test never showed. "I want to know our test would have caught that."

Darwaza has no answer to either yet: it has never been tested over HTTP, never been pushed past Tuesday's traffic, and the replay's report is one number for six brands.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    over-http.ts: Darwaza over HTTP

    Write and export start(port): serve gateway (from ./darwaza.ts) on that port with serve() from ../../shared/jawab.ts, and return the server.

    Under if (import.meta.main), start(8700), replay 'http://127.0.0.1:8700' from 10:00 to 12:00 with the label 'Darwaza over HTTP', printReport it, then close the server (server.close() and server.closeAllConnections()).

  2. Task 2

    closed-loop.ts: open loop, closed loop

    Write and export closedLoop(handle, entries, users): users workers at once, each taking the next entry not yet sent, awaiting handle(entry.request) and timing it, until none are left. Return { requests, answered (% ok, one decimal), p50, p95 } (percentile() of the times).

    Under if (import.meta.main), with v0 from ./darwaza.ts: replay(v0, { from: '13:50', to: '14:50' }) and print open loop: N requests, answered X%, p50 N ms, p95 N ms (answered is successRate); then startDay('13:50') (given), closedLoop(v0, the same entries, 2), resetClock(), and print the same line starting closed loop:.

  3. Task 3

    capacity.ts: how much can Darwaza take?

    Write and export meetsSlo(report): successRate at least 99.5 and p95 at most 1,000.

    Under if (import.meta.main), for msPerMinute 25, 12, 6, 3 in turn, replay the whole day through a new createDarwaza({ semantic: 0.85 }) at that speed and print M ms a minute: R requests a second, answered X%, p95 N ms, SLO met (or missed), where R is 626 / (720 x M / 1000), rounded. Stop after the first speed that misses the SLO, then print capacity: about R requests a second (the fastest speed that met it).

  4. Task 4 · interview question

    Interview: team-report.ts, does every brand get the SLO?

    Write and export teamReport(results): one row per team that has results, in the order of TEAMS: { team, requests, refused, answered (% ok of those not refused, one decimal), p95 (percentile() of their ms), inr (the sum of costInr, missing counts 0, two decimals) }.

    Under if (import.meta.main), replay the whole day through createDarwaza({ semantic: 0.85 }) and print a line per row (TEAM N requests N refused answered X% p95 N ms ₹Y), then all of Jawab: answered X%, p95 N ms, then SLO missed by: TEAM, TEAM (under 99.5% or over 1,000 ms) or SLO met by every team.

What you will learn

  • Testing what users meet: the gateway over HTTP.
  • Open loop and closed loop, and coordinated omission: how a load test can hide exactly the slowness it exists to find.
  • Capacity: speeding up the day until the SLO breaks, and what breaks first.
  • The SLO per team, and why small numbers are harsh.

More in Run it in production

Build "Load testing the gateway" yourself

Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build an AI gateway (TypeScript) are free. This one comes with the all labs plan.