AI engineering traps · Chapter 9 of 16
Counting tokens with string length
Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Calling the model
- 3 tasks
- Python
- In the all labs plan
The first 2 chapters of AI engineering traps are free. This one comes with the all labs plan.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
repro.py: break Gomti Heights' digest on purpose
Write reproduce() in repro.py. Ask digest from digest.py for the digest of 'Gomti Heights', with today's messages from whatsapp.py (gomti_heights_today()). Return the digest, or, if digest raises an openai.APIStatusError e, the error's status and message: f"{e.status_code} {e.body['message']}".
The check runs your reproduce() twice: beside the original digest.py it must return the 400, "400 This model's maximum context length is 8192 tokens. However, you requested 12560 tokens (12060 in the messages, 500 in the completion). Please reduce the length of the messages or completion.", and beside a fixed digest.py 'Digest for Gomti Heights: 309 messages. Most talked about: water (92), cleaning (31).' digest uses the lab model when you leave out its client; reproduce() must finish within 10 seconds.
- Task 2
digest.py: count tokens the way the model does
Fix fit_messages in digest.py so it measures each line the way the lab model does: a line sent as a message of its own costs its tokens in the o200k tokenizer plus 3. It keeps the newest lines, in their order, whose costs add up to at most budget (BUDGET, 6000, when left out), and no fewer than fit. tiktoken is installed: tiktoken.get_encoding("o200k_base") is the lab model's own tokenizer.
Keep BUDGET, digest and its printed line digest: <kept> of <n> messages sent. The check uses lines and budgets of its own (Hindi, English, emoji), and a big Hindi day with the lab model. Dividing the length by 2 instead of 4 is not a fix. whatsapp.py is not to be changed.
- Task 3
test_digest.py: a test that would have caught it
Write a pytest file, test_digest.py, that would have caught this bug. The check runs your file three times: beside the original digest.py, where at least one test must fail; beside the fixed digest.py, where every test must pass; and beside a wrong fix that guesses 2 characters a token, where at least one test must fail.
So, for both groups in whatsapp.py, take the lines digest would send ('[time] name: text') and check fit_messages with the real tokenizer (tokens plus 3 a line): what it keeps costs at most BUDGET, and one more line would not fit. Each test must finish within 10 seconds.
What you will learn
- repro.py: break Gomti Heights' digest on purpose
- digest.py: count tokens the way the model does
- test_digest.py: a test that would have caught it
More in Calling the model
Build "Counting tokens with string length" yourself
Sunwai, a Lucknow app for housing societies, runs on a model, and Farah, its CTO, keeps assigning you tickets. Each is a trap AI engineering interviews love to ask about. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of AI engineering traps are free. This one comes with the all labs plan.