Building with LLMs · Chapter 4 of 24
Sampling
By the end of this chapter you can control how a model picks its words: make its answers repeatable or varied, cut them short on purpose, ask for several at once, and choose the right settings for each job. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Talking to a model
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Building with LLMs are free. This one comes with the all labs plan.
Why you need this
Lata tried the bot on one refund question five times and got five slightly different answers. For a customer reply that is fine, even good: nobody wants a robot that says the same sentence to everyone. But Nikhil is writing tests for the bot, and a test that passes on one run and fails on the next is useless. And Lata wants something else: three drafts to choose from, not one.
All of this is decided by a handful of request settings you have not used yet. They do not change what the model knows. They change how it picks each token from the probabilities it computes (chapter 1).
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
temperatures.py: how much answers vary
Nikhil wants to see what temperature does. Ask lab-chat, as one user message:
Can I bring my child on the bus?
three times: at temperature 0, at temperature 1 and at temperature 1.8, each request asking for 4 answers at once.
For each request, in that order, print how many of its 4 answers are different from each other, like this:
temperature 0.5: 2 of 4 different
- Task 2
limits.py: two ways to stop early
Lata wants short replies, and needs to know when one was cut off. Ask lab-chat, as one user message:
Can I change my travel date?
twice: first with a limit of 10 tokens on the reply, then with no limit but stopping at the first comma.
For each, print the reply, a space, a vertical bar, a space and the reply's finish_reason, like this:
Hello there | stop
- Task 3
seeded.py: a draft you can repeat
Nikhil wants varied drafts that a test can still repeat. Write a function draft(question, seed) in seeded.py that asks lab-chat the question, as one user message, at temperature 1 with that seed, and returns the reply's text.
The check calls it twice with the same question and seed (the two drafts must be the same) and with other seeds, and reads the requests.
- Task 4 · interview question
Interview: fix_me.py, Lata's four drafts
fix_me.py should ask lab-chat, in one request, for 4 drafts of a reply at temperature 0.9 and print each draft in full, one per line, in the order the API gives them. It has three mistakes. Run it, read what goes wrong, fix it, and run it again.
Keep the question and seed=4 exactly as they are.
What you will learn
- temperature: how adventurous the pick is, from 0 (always the most likely) to 2.
- top_p: leaving out the unlikely tokens altogether.
- n and seed: several answers from one request, and repeatable ones.
- max_tokens and stop: ending a reply early, and how finish_reason tells you which happened.
- When to use 0, and when not to.
More in Talking to a model
Build "Sampling" yourself
SafarLine, an intercity bus company, drowns in WhatsApp messages. You build its first AI assistant. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Building with LLMs are free. This one comes with the all labs plan.