OpenAI Agents SDK · Chapter 2 of 24
Model settings
By the end of this chapter you can say how a model picks its words and what temperature and top_p change, give an agent model settings and read them in the request the SDK sends, cap a reply's length and handle a reply cut short, set the model and settings for a whole run with a RunConfig, and work out what a run costs in rupees. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Your first agents
- 4 tasks
- Python
- Free chapter
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.
Why you need this
Bhanu tried the chapter 1 agent on the parents' line for an afternoon and came back with two complaints. The same question got slightly different replies each time ("Sure. ...", "Happy to help. ..."), which makes Shikhar sound unsure of itself. And one evening run added a sentence nobody asked for. Nidhi has a third question: the helpdesk will answer about 2,000 messages a day, so what will that cost, and would the cheap model do?
An agent from chapter 1 has a name, instructions and a model. This chapter adds the knobs on the model itself: how freely it writes, how long it may write, and which model a run uses. They do not make the answers true (the model still has not seen Shikhar's policy), but they make it predictable and priced.
The lesson
How a model picks its words
A model writes one token at a time. For each next token it works out a probability for every token it knows ("The" 40%, "Sure" 25%, "Happy" 10% ...) and then samples one: it picks at random, weighted by those probabilities. That is why the same question can come back worded a little differently. Here is one agent from chapter 1 run three times on the same question (one run; yours will differ):
agent = Agent(name="Shikhar helpdesk", instructions="You are Shikhar's helpdesk. Be brief.", model="lab-chat")
for _ in range(3):
print(Runner.run_sync(agent, "When is the weekly test?").final_output)
Sure. The weekly test is on Saturday at 2 pm.
Happy to help. The weekly test is on Saturday at 2 pm.
The weekly test is on Saturday at 2 pm.
Two settings change the sampling:
- temperature (0 to 2, default 1) sharpens or flattens the probabilities. At 0 the model always takes the most likely token, so the same request gets the same reply. Above 1 unlikely tokens get more chances.
- top_p (0 to 1, default 1) keeps only the most likely tokens whose probabilities add up to p, and samples from those. At 0.9 the unlikely tail is cut off.
Change one of them, not both, and only for a reason.
ModelSettings on an agent
The SDK keeps these knobs in a ModelSettings object, and an agent takes one as model_settings=, next to its model:
agent = Agent(
name="Shikhar helpdesk",
instructions="You are Shikhar's helpdesk. Be brief.",
model="lab-chat",
model_settings=ModelSettings(temperature=0),
)
The weekly test is on Saturday at 2 pm.
The weekly test is on Saturday at 2 pm.
The weekly test is on Saturday at 2 pm.
Every setting you give becomes a field of the Responses request, and a setting you leave out is not sent at all (the model uses its own default). A copy of the agent at temperature=1.5, top_p=0.9, made with clone(), sends this (lab requests 1, trimmed):
"request": {
"model": "lab-chat",
"input": [{"content": "When is the weekly test?", "role": "user"}],
"instructions": "You are Shikhar's helpdesk. Be brief.",
"temperature": 1.5,
"top_p": 0.9,
"tools": []
}
and answered, over three runs:
Happy to help. The weekly test is on Saturday at 2 pm.
Of course! The weekly test is on Saturday at 2 pm. By the way, the JEE syllabus has been cut by half this year.
Sure. The weekly test is on Saturday at 2 pm.
A hot model adds things. For a helpdesk, keep temperature low. ModelSettings has more fields (tool_choice is chapter 6's); this chapter uses three.
A cap on the reply: max_tokens
max_tokens is the most output tokens the model may write. The SDK sends it as the request's max_output_tokens. A WhatsApp reply should be short, but a cap is a hard stop, not a request to be brief: the model writes until it reaches it, mid-sentence if need be.
agent = Agent(name="Shikhar helpdesk", model="lab-chat", model_settings=ModelSettings(max_tokens=5))
result = Runner.run_sync(agent, "Can my son change his hostel room?")
agents.exceptions.ModelBehaviorError: Responses stream ended with terminal event `response.incomplete`. status=incomplete; incomplete_details=IncompleteDetails(reason='max_output_tokens').
That is the last line of the traceback. The SDK also logs "Error getting response" on stderr. In the request log the response says why:
"status": "incomplete",
"incomplete_details": {"reason": "max_output_tokens"},
"output": [{"type": "message", "content": [{"type": "output_text", "text": "Happy to help. Hostel"}]}]
The model did answer, but only "Happy to help. Hostel". The SDK will not hand that to you as a final output: a reply that stopped in the middle is not an answer, so it raises ModelBehaviorError, the error for "the model's response cannot be used". You can import it from agents and catch it with try and except like any Python exception, then decide what to do: retry with more room, or tell the parent to wait. To keep replies short, say so in the instructions, and keep max_tokens as a safety cap well above a normal reply.
One run, every agent: RunConfig
Settings on an agent travel with that agent. Sometimes you want one run to override them all: try every desk on the cheap model for a day, or run a test at temperature 0. A RunConfig holds settings for one run, and you pass it to the Runner:
fees = Agent(name="Fees desk", model="lab-chat",
model_settings=ModelSettings(temperature=0.7, max_tokens=300))
hostel = Agent(name="Hostel desk", model="lab-chat-large")
config = RunConfig(model="lab-chat-mini", model_settings=ModelSettings(temperature=0))
print(Runner.run_sync(fees, "Is there a scholarship for a good SSAT score?", run_config=config).final_output)
print(Runner.run_sync(hostel, "Are meals included in the hostel?", run_config=config).final_output)
The two requests, from lab requests 2 (trimmed):
"model": "lab-chat-mini", "temperature": 0, "max_output_tokens": 300
"model": "lab-chat-mini", "temperature": 0
- The RunConfig's model wins over every agent's model for that run.
- Its model settings are merged field by field: the fees desk's temperature 0.7 became 0, but its
max_tokens=300stayed, because the RunConfig did not set one. - The agents are not changed: the next run without the RunConfig uses lab-chat and lab-chat-large again.
What a run costs
lab models in the terminal lists the lab's models and their prices, in rupees per million tokens (the first three of the list):
lab-chat window 8192 ₹200 in, ₹800 out, per million tokens
lab-chat-mini window 8192 ₹15 in, ₹60 out, per million tokens
lab-chat-large window 131072 ₹1000 in, ₹4000 out, per million tokens
Output tokens cost four times input tokens, and lab-chat-mini is more than ten times cheaper than lab-chat. A run's price is its input tokens times the input price plus its output tokens times the output price, divided by a million:
usage = Runner.run_sync(agent, "Are meals included in the hostel?").context_wrapper.usage
print(usage.input_tokens, usage.output_tokens)
rupees = (usage.input_tokens * 200 + usage.output_tokens * 800) / 1_000_000
print(f"₹{rupees:.4f} a reply")
27 18
₹0.0198 a reply
About two paise. Small, until it is multiplied by every message of every day, and by every request of a run (agents with tools make several, from chapter 4). The cheap model is not always the saving it looks like: a wrong answer about a refund costs a family far more than the difference.
Your turn
Four tasks. Task 1 gets the same reply twice at temperature 0. Task 2 caps a reply too tightly, catches the ModelBehaviorError and runs again with room. Task 3 runs two lab-chat desks on lab-chat-mini with one RunConfig. Task 4 is an interview question: price a day of 2,000 messages on both models, and be ready to say when the cheap model is the wrong saving.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
steady.py: the same question, the same reply
Make an agent for lab-chat with the instructions You are Shikhar Classes' helpdesk assistant. and model settings with a temperature of 0. Run it twice on the parent's question exactly:
Can I pay the fees in instalments?
Print the two final outputs, one per line, then a third line: Same: and whether the two are equal (True or False).
- Task 2
too_short.py: a reply cut short
Make an agent for lab-chat, with no instructions, whose model settings allow at most 8 output tokens, and run it on:
What time do students have to be back in the hostel?
The reply is cut short and the SDK raises ModelBehaviorError. Catch that error (that class, not every exception) and print: Cut short at 8 tokens, with the 8 read from the agent's model settings. Then make a copy of the agent with model settings allowing 200 tokens, run it on the same question and print its final output.
- Task 3
run_config.py: one setting for the whole run
Make two agents, both for lab-chat: Admissions desk, with the instructions You are the admissions desk of Shikhar Classes., and Academics desk, with You are the academics desk of Shikhar Classes.
Make one RunConfig with the model lab-chat-mini and model settings with a temperature of 0. Run the admissions desk on When does the next NEET batch start? and then the academics desk on When is the weekly test?, both with that RunConfig. Print one line per run: the agent's name, a colon and a space, then the final output. Keep model="lab-chat" on both agents and do not clone them.
- Task 4 · interview question
Interview: day_cost.py, what a day costs
Make an agent for lab-chat with the instructions You are Shikhar Classes' helpdesk assistant. Answer in one or two sentences. and its clone for lab-chat-mini. Run the clone, then the agent, on the parent's message:
fees 10 din late ho gayi, kitna fine lagega?
From each run's usage work out the price of one reply, then of 2,000 replies, with the prices in rupees per million tokens: lab-chat-mini 15 in and 60 out, lab-chat 200 in and 800 out. Print one line per model: the model's name, a colon and a space, the rupee sign and the price with two decimals, then a day (for example: lab-chat: ₹39.20 a day).
The interviewer will ask when the cheap model is the wrong saving.
What you will learn
- How a model picks its words: temperature and top_p.
- ModelSettings on an agent, and what they become in the request.
- max_tokens: a cap on the reply, and the error when a reply is cut short.
- RunConfig: one model and one set of settings for a whole run.
- The price of a run, from its usage and the lab's price list.
More in Your first agents
Build "Model settings" yourself
Shikhar Classes, a Kota coaching institute, answers parents and students on WhatsApp all day. You build its helpdesk assistant. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.