AI engineering traps · Chapter 5 of 16

One history for everyone

Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Calling the model
  • 3 tasks
  • Python
  • In the all labs plan

The first 2 chapters of AI engineering traps are free. This one comes with the all labs plan.

Why you need this

The model does what it is told with what it is sent. When another resident's messages are in the request, they are part of "the conversation" as far as the model knows, and "what is my flat?" is answered from the last flat anyone mentioned. The model is not breaking a rule: your code handed it someone else's data. An instruction in the prompt ("only talk about this resident") cannot fix that; only not sending it can.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    repro.py: tell Mr. Pandey someone else's flat on purpose

    Write reproduce() in repro.py. With answer from assistant.py, have two residents chat, in this order, word for word: first resident 'R-1042' says My flat is B-304 and my phone is 98390 12345., then resident 'R-2213' asks What is my flat number and phone number? Return the reply R-2213 gets.

    The check runs your reproduce() twice: beside the original assistant.py it must return 'Your flat is B-304 and your phone number is 98390 12345.' (Mr. Srivastava's details, given to Mr. Pandey), and beside a fixed assistant.py "I don't have your flat number yet. Please tell me your flat number and phone number." answer uses the lab model when you leave out its client; reproduce() must finish within 10 seconds.

  2. Task 2

    assistant.py: give every resident a history of their own

    Fix answer in assistant.py so each resident has a history of their own, found by resident_id: a resident's request holds the system message and only that resident's own earlier messages and replies, then the new message. A resident must still be remembered from one message to the next.

    Keep answer(resident_id, text, client=None) and the printed line sahayak: <resident_id>, <n> messages, where n is that resident's history after the reply. The check chats as several residents in turns and reads every request. Clearing the history before every message is not a fix.

  3. Task 3

    test_assistant.py: a test that would have caught it

    Write a pytest file, test_assistant.py, that would have caught this bug. The check runs your file three times: beside the original assistant.py, where at least one test must fail; beside the fixed assistant.py, where every test must pass; and beside a wrong fix that clears the history before every message, where at least one test must fail.

    So have two residents chat with the lab model: one gives their flat and phone, the other asks for theirs and must not get the first one's; then the first asks again and must get their own. Each test must finish within 10 seconds.

What you will learn

  • repro.py: tell Mr. Pandey someone else's flat on purpose
  • assistant.py: give every resident a history of their own
  • test_assistant.py: a test that would have caught it

More in Calling the model

Build "One history for everyone" yourself

Sunwai, a Lucknow app for housing societies, runs on a model, and Farah, its CTO, keeps assigning you tickets. Each is a trap AI engineering interviews love to ask about. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI engineering traps are free. This one comes with the all labs plan.