Python traps · Chapter 2 of 16

Late-binding closures in loops

Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Everyday traps
  • 3 tasks
  • Python
  • Free chapter

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.

Why you need this

Called inside the loop, the same lambda is right every time, because rider still holds the current rider:

calls = []
for rider in ["Santosh", "Imran", "Prakash"]:
    calls.append(lambda: f"calling {rider}")
    print(calls[-1]())
calling Santosh
calling Imran
calling Prakash

So the code was fine until the calls moved later. Nothing raises: each call finds a rider, just the wrong one. Ruff, in the lab, knows the trap (trimmed):

$ ruff check --select B023 calls.py
B023 Function definition does not bind loop variable `rider`
 --> calls.py:4:41

The usual fix is to bind the value when the function is made. A default argument is worked out when the lambda line runs, so each lambda keeps its own rider. With this line in make_calls, the first line the example prints becomes:

calls.append(lambda rider=rider: f"calling {rider}")
['calling Santosh', 'calling Imran', 'calling Prakash']

functools.partial(greet, rider) does the same with a named function, and so does a small function that takes the rider and returns the lambda. Careful what you bind: a function that keeps a place in a list and reads the list when it runs is late again, as soon as someone changes the list.

The lesson

By the end of this case

You can tell a late-binding closure from its symptoms, make it happen on purpose by running functions after the loop that made them, fix it by binding each value when the function is made, and guard it with a test that runs the jobs late, the way production does.

The ticket

DD-415: Five customers, one name in the SMS. Assigned to you by Kunal. Priority: high. Since: Thursday's release.

Bhavna, in the riders group at 12:40 pm: "The 12:30 batch had five tiffins. Every customer got an SMS saying 'Hi Myra, your tiffin D2135 is on its way'. Neel in Andheri called us asking who Myra is, and Yash thought his order was cancelled. Only Myra got the right one."

The SMS sender's log for the 12:30 batch:

12:31:02  notify  sms: Hi Myra, your DabbaDrop tiffin D2135 is on its way.
12:31:02  notify  sms: Hi Myra, your DabbaDrop tiffin D2135 is on its way.
12:31:03  notify  sms: Hi Myra, your DabbaDrop tiffin D2135 is on its way.
12:31:03  notify  sms: Hi Myra, your DabbaDrop tiffin D2135 is on its way.
12:31:03  notify  sms: Hi Myra, your DabbaDrop tiffin D2135 is on its way.

Kunal: "Thursday's release moved the SMS to a queue. Before, we sent each text inside the loop over the batch; now build_jobs makes the jobs and the sender runs them a little later. D2135 was the last order in that batch. Reproduce it, fix it, and make sure it never comes back. You are in notify.py."

A function looks its names up when it runs

A lambda (or a def) inside a loop does not copy the values it uses. It keeps the variable and reads it when it is called. Here is a rider call list of the same kind:

def make_calls(riders):
    calls = []
    for rider in riders:
        calls.append(lambda: f"calling {rider}")
    return calls


calls = make_calls(["Santosh", "Imran", "Prakash"])
print([call() for call in calls])
print(calls[0].__closure__[0].cell_contents)
print(calls[0].__closure__[0] is calls[2].__closure__[0])
['calling Prakash', 'calling Prakash', 'calling Prakash']
Prakash
True

A function that uses a variable from the function around it is a closure. It holds that variable in a cell (__closure__), and all three lambdas share the one cell for rider. When the loop ends, the cell holds the last rider, so every call says Prakash.

Making it happen on purpose

The bug only shows when the jobs run after the loop. Your reproduce() builds the jobs for two orders, then runs them, and returns the texts. The check runs it beside the original notify.py, where both texts must be the last order's, and beside a fixed one, where each must be its own.

A test that runs the jobs late

Your guard does what production does, in this order: build the jobs, then clear the batch list and fill it with the next slot's orders, and only then run the jobs and check each text. A test that runs each job straight away passes the bug, and one that never touches the list passes a fix that reads the list late.

Your turn

Three tasks, one file each. In repro.py, send Neel Myra's SMS on purpose, and run it with python3 repro.py. In notify.py, fix build_jobs so each job keeps its own order. In test_notify.py, write the guard and run it with python3 -m pytest test_notify.py: the check runs it beside the original code, the fix and a wrong fix that reads the batch list late. Then answer Kunal's interview question at the bottom of the case.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    repro.py: two customers, one name

    Write reproduce() in repro.py. Make a batch of two orders, {"id": "D2131", "name": "Neel"} and {"id": "D2135", "name": "Myra"}, give it to build_jobs from notify.py, then run the jobs it gave with run_jobs, and return the list of texts run_jobs gives.

    The check runs your reproduce() twice: beside the original notify.py it must return Myra's text twice, ["Hi Myra, your DabbaDrop tiffin D2135 is on its way.", "Hi Myra, your DabbaDrop tiffin D2135 is on its way."], and beside a fixed notify.py it must return Neel's text, "Hi Neel, your DabbaDrop tiffin D2131 is on its way.", then Myra's. So it must really build the jobs and run them. No real waiting and no input(): reproduce() must finish within 3 seconds.

  2. Task 2

    notify.py: every customer gets their own SMS

    Fix build_jobs(batch) in notify.py. It must give a list with one job per order, in the batch's order, and each job is a function called with no arguments that gives that order's own text, the same as sms_text(order). A job must give the same text whenever it runs: right after build_jobs, in reverse order, a second time, and after the caller has cleared the batch list or filled it with the next slot's orders. An empty batch gives an empty list.

    Keep sms_text, run_jobs and run_jobs' log line (sms: <text>) as they are. The check imports a fresh copy of your notify.py and uses its own orders, not D2131, and a job that keeps its order's place in the list and reads the list when it runs is as wrong as the bug: after a refill it texts the next slot's customer.

  3. Task 3

    test_notify.py: a test that would have caught it

    Write a pytest file, test_notify.py, that would have caught this bug before the 12:30 batch. The check runs your file three times: beside the original notify.py, where at least one test must fail; beside the fixed notify.py, where every test must pass; and beside a wrong fix whose jobs keep their order's place in the batch list and read the list when they run, where at least one test must fail.

    So test it the way production runs: jobs for two or more different orders each give their own text, and jobs built for a batch still give their own texts after the batch list is cleared and filled with other orders. No real waiting: each test must finish within 3 seconds.

What you will learn

  • repro.py: two customers, one name
  • notify.py: every customer gets their own SMS
  • test_notify.py: a test that would have caught it

More in Everyday traps

Build "Late-binding closures in loops" yourself

DabbaDrop runs on Python, and Kunal, its senior developer, keeps assigning you tickets. Each is a real trap interviews love to ask about. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.