AI engineering traps · Chapter 12 of 16

Results matched by finishing order

Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Tools
  • 3 tasks
  • Python
  • In the all labs plan

The first 2 chapters of AI engineering traps are free. This one comes with the all labs plan.

Why you need this

Each tool result goes back with a tool_call_id (case 11), and the model trusts it: the result for B-304's call is what B-304 owes. The results here say only {"dues":1200}, so when the amounts are matched to the calls by position in a list that was filled in finishing order, the model has no way to notice. It writes a confident, wrong sentence.

Running the lookups one after another would also get the amounts right, by giving up the speed. The fix keeps them parallel and keeps each result with its own call.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    repro.py: let C-12's lookup finish first, on purpose

    Write reproduce() in repro.py. Build fake services, an object whose async dues(flat) gives 4500 for 'B-304', but only after one turn of the event loop (await asyncio.sleep(0)), and 1200 for any other flat straight away. Ask ask_treasurer from treasurer.py 'Compare the dues of B-304 and C-12' with your fake, with asyncio.run, and return its answer.

    The check runs your reproduce() twice: beside the original treasurer.py it must return 'B-304 owes ₹1,200. C-12 owes ₹4,500.' (the swapped answer in the ticket), and beside a fixed treasurer.py 'B-304 owes ₹4,500. C-12 owes ₹1,200.' No real waiting: one turn of the event loop is enough. reproduce() must finish within 10 seconds.

  2. Task 2

    treasurer.py: keep each result with its own call

    Fix ask_treasurer in treasurer.py. Still run all the dues lookups at the same time (every lookup starts before any finishes), but send each result back with the id of the call that asked for it, whatever order the lookups finish in.

    Keep the tool, the printed line treasurer: <n> lookups in parallel and the answer as they are. The check asks about three flats of its own with the lab model, holds the lookups back so they finish in the opposite order to the calls, and watches when they start. Running the lookups one at a time is not the fix.

  3. Task 3

    test_treasurer.py: a test that would have caught it

    Write a pytest file, test_treasurer.py, that would have caught this race before the treasurer saw it. The check runs your file three times: beside the original treasurer.py, where at least one test must fail; beside the fixed treasurer.py, where every test must pass; and beside a wrong fix that runs the lookups one at a time, where at least one test must fail.

    So use fake services where the first flat's lookup finishes last, and assert each flat's own amount in the answer; and record when each lookup starts and finishes, and assert both start before either finishes. Use a plain def test that calls asyncio.run. Each test must finish within 10 seconds.

What you will learn

  • repro.py: let C-12's lookup finish first, on purpose
  • treasurer.py: keep each result with its own call
  • test_treasurer.py: a test that would have caught it

More in Tools

Build "Results matched by finishing order" yourself

Sunwai, a Lucknow app for housing societies, runs on a model, and Farah, its CTO, keeps assigning you tickets. Each is a trap AI engineering interviews love to ask about. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI engineering traps are free. This one comes with the all labs plan.