AI engineering traps · Chapter 3 of 16

Temperature and a flaky test

Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Calling the model
  • 3 tasks
  • Python
  • In the all labs plan

The first 2 chapters of AI engineering traps are free. This one comes with the all labs plan.

Why you need this

A seed makes the draw repeatable: same request, same seed, same reply. With seed=3, that question gets "Happy to help. Maintenance dues..." on every ask. Repeatable is not the same as right. With a seed and the default temperature, some complaints get their bare label every time, and others get "Sure. parking" every time. The flaky test becomes a steady wrong answer. (Real providers also say a seed is "best effort", not a promise.)

When the reply is a decision your code acts on (a label, a yes or no, a choice from a list), ask with temperature 0, and still check the reply against the list, as this code already does.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    repro.py: make the label change on purpose

    Write reproduce() in repro.py. Ask classify from labels.py to label the ticket's complaint 20 times, word for word: The lift in Block C is stuck again and the alarm is not working. Return the different labels it gave, each once, sorted: a list of str.

    The check runs your reproduce() twice: beside the original labels.py it must return ['lift', 'other'] (the label changes from ask to ask), and beside a fixed labels.py ['lift']. Twenty asks are enough for the bad luck to show every time. classify uses the lab model when you leave out its client; reproduce() must finish within 10 seconds.

  2. Task 2

    labels.py: make the label the same every time

    Fix classify in labels.py so the same complaint always gets the same, bare label: ask the lab model with temperature 0. Keep the rest as it is: a reply that is not one of the seven labels is still 'other', and keep the printed line label: '<reply>' -> <category>.

    The check labels complaints of its own, each three times, and looks at every request classify sends: each must have temperature 0. A seed alone is not a fix: it makes a reply repeatable, not right.

  3. Task 3

    test_labels.py: a test that would have caught it

    Write a pytest file, test_labels.py, that fails on this bug every run, not some runs. The check runs your file twice: beside the original labels.py, where at least one test must fail, and beside the fixed labels.py, where every test must pass.

    So ask for the same complaint's label many times (20) and assert it every time, or hand classify a fake model that remembers the request and check its temperature. Each test must finish within 10 seconds.

What you will learn

  • repro.py: make the label change on purpose
  • labels.py: make the label the same every time
  • test_labels.py: a test that would have caught it

More in Calling the model

Build "Temperature and a flaky test" yourself

Sunwai, a Lucknow app for housing societies, runs on a model, and Farah, its CTO, keeps assigning you tickets. Each is a trap AI engineering interviews love to ask about. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI engineering traps are free. This one comes with the all labs plan.