Build an AI gateway · Chapter 19 of 21

Load testing the gateway

By the end of this chapter you can load-test Darwaza the way Jawab's app will use it, over HTTP; tell an honest load test from one that hides the worst minutes; find how much traffic Darwaza can take before the SLO breaks; and check the SLO for each brand, not only for Jawab as a whole. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Run it in production
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Build an AI gateway are free. This one comes with the all labs plan.

Why you need this

Every chapter so far measured Darwaza by calling its handle function in the same program as the replay. Jawab's app will not do that: it will send HTTP requests to Darwaza running as a service. And Tuesday is only one day. Anjali has signed two more brands for next month, and Payyoli is planning a bigger sale. Her question is simple: "How much more can Darwaza take before customers notice?"

Joseph has a second question. A friend's team load-tested their gateway with a popular tool, saw p95 under 600 ms, and went live; the first outage gave customers six-second waits the test never showed. "I want to know our test would have caught that."

Darwaza has no answer to either yet: it has never been tested over HTTP, never been pushed past Tuesday's traffic, and the replay's report is one number for six brands.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    over_http.py: Darwaza over HTTP

    Write async def start(port) -> asyncio.Server: serve gateway (from darwaza) on that port with await serve(gateway, port) from jawab, and return the server.

    Under if __name__ == "__main__":, server = await start(8700), replay "http://127.0.0.1:8700" from "10:00" to "12:00" with the label "Darwaza over HTTP", print_report it, then close the server (server.close() and await server.wait_closed()).

  2. Task 2

    closed_loop.py: open loop, closed loop

    Write async def closed_loop(handle, entries, users) -> dict: users coroutines at once (asyncio.gather), each taking the next entry not yet sent, awaiting handle(entry.request) and timing it, until none are left. Return {"requests", "answered" (% ok, one decimal), "p50", "p95"} (percentile() of the times).

    Under if __name__ == "__main__":, with v0 from darwaza: replay(v0, "13:50", "14:50") and print open loop: N requests, answered X%, p50 N ms, p95 N ms (answered is success_rate); then start_day("13:50") (given), closed_loop(v0, the same entries, 2), reset_clock(), and print the same line starting closed loop:.

  3. Task 3

    capacity.py: how much can Darwaza take?

    Write def meets_slo(report) -> bool: success_rate at least 99.5 and p95 at most 1,000.

    Under if __name__ == "__main__":, for ms_per_minute 25, 12, 6, 3 in turn, replay the whole day through a new create_darwaza(semantic=0.85) at that speed and print M ms a minute: R requests a second, answered X%, p95 N ms, SLO met (or missed), where R is 626 / (720 x M / 1000), rounded. Stop after the first speed that misses the SLO, then print capacity: about R requests a second (the fastest speed that met it).

  4. Task 4 · interview question

    Interview: team_report.py, does every brand get the SLO?

    Write def team_report(results) -> list[dict]: one row per team that has results, in the order of TEAMS: {"team", "requests", "refused", "answered" (% ok of those not refused, one decimal), "p95" (percentile() of their ms), "inr" (the sum of cost_inr, None counts 0, two decimals)}.

    Under if __name__ == "__main__":, replay the whole day through create_darwaza(semantic=0.85) and print a line per row (TEAM N requests N refused answered X% p95 N ms ₹Y), then all of Jawab: answered X%, p95 N ms, then SLO missed by: TEAM, TEAM (under 99.5% or over 1,000 ms) or SLO met by every team.

What you will learn

  • Testing what users meet: the gateway over HTTP.
  • Open loop and closed loop, and coordinated omission: how a load test can hide exactly the slowness it exists to find.
  • Capacity: speeding up the day until the SLO breaks, and what breaks first.
  • The SLO per team, and why small numbers are harsh.

More in Run it in production

Build "Load testing the gateway" yourself

Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build an AI gateway are free. This one comes with the all labs plan.