Building with LLMs · Chapter 1 of 24

What an LLM is

By the end of this chapter you can call a language model from Python, read what comes back, explain why its answers vary and why it sometimes invents things, and fix the errors the API sends you. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Talking to a model
  • 4 tasks
  • Python
  • Free chapter

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.

Why you need this

SafarLine runs buses between seven cities, and its customers write on WhatsApp all day: is there a seat to Goa tonight, how much luggage can I take, where is my refund. Lata and one other agent answer every message by hand, and the queue is longer each evening. Zoya, the founder, wants a model to draft the answers.

Everything you know from Python so far works on data you can see: you read a file, you compute, you get the same result every time. A model is different. It writes text nobody wrote before, it can give two different answers to one question, and it can state something false with complete confidence. Before SafarLine lets it near a customer, you need to know what it is, how to call it, and what it does when it does not know.

The lesson

A model predicts the next token

A large language model (LLM) is a program trained on a huge amount of text to do one thing: given some text, predict what comes next. It does not look things up. It works on tokens, pieces of text of a few characters each (chapter 3 counts them), and it writes an answer one token at a time:

  1. It reads everything it was sent.
  2. It gives every token it knows a probability of coming next.
  3. It picks one (more on how in a moment), adds it to the text, and goes back to step 1, until it decides the answer is finished.

Training is what made those probabilities good: after reading billions of sentences, "The bus leaves at" is very likely followed by a time. That is also the model's limit. It knows what text usually says, not what is true for SafarLine, which it has never seen.

The lab model and the openai package

Models run on a provider's servers, and you reach them over HTTP. The openai package is the official Python client for OpenAI's API, and many providers accept the same API. This lab has its own model, lab-chat, running inside your lab with the same API, so you need no account and no internet. Two environment variables point the package at it:

$ echo $OPENAI_BASE_URL
http://127.0.0.1:11500/v1

OPENAI_API_KEY is set too. Point those two at a real provider, with your own key, and the same code talks to a real model. Here is a call in the REPL:

>>> from openai import OpenAI
>>> client = OpenAI()
>>> response = client.chat.completions.create(
...     model="lab-chat",
...     messages=[{"role": "user", "content": "What does PNR mean?"}],
... )
>>> response.choices[0].message.content
'A PNR, or Passenger Name Record, is the code that identifies a booking.'
  • OpenAI() reads the address and the key from the environment.
  • A request names the model and sends a list of messages. Each message is a dictionary with a role ("user" is the customer speaking) and its content. Chapter 2 is all about roles.
  • The answer is not the response itself: it is in choices[0].message.content.

The response object

Print the whole response and you see what the API sends back:

ChatCompletion(id='chatcmpl-laba39ac3dfe6f121dda046', choices=[Choice(
finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(
content='Sure. Small pets are welcome on board in a carrier.', role='assistant',
...)], created=1792516627, model='lab-chat', object='chat.completion',
usage=CompletionUsage(completion_tokens=12, prompt_tokens=15, total_tokens=27, ...))
  • choices is a list, because you can ask for several answers at once (chapter 4). Most of the time there is one, choices[0].
  • message is the model's turn: its role is "assistant" and its content is the text.
  • finish_reason says why it stopped: "stop" means it finished; "length" means it hit a limit you set.
  • usage counts tokens: what you sent (prompt_tokens) and what it wrote (completion_tokens). Providers charge for both.

Why answers vary, and why models invent

That answer was to "Can I bring my dog on the bus?". Ask three times:

'Small pets are welcome on board in a carrier.'
'Good question. Small pets are welcome on board in a carrier.'
'Sure. Small pets are welcome on board in a carrier.'

The wording changes because of step 3: the model does not always take the most likely token, it samples from the probabilities, so a less likely first word ("Good", "Sure") sometimes wins. Chapter 4 controls this.

The content is worse: SafarLine does not carry pets, except guide dogs (policy, section 6.3). The model did not refuse or say it does not know. It produced what bus companies' rules often say, fluently, because that is what prediction does. This is called a hallucination, and it is not a bug you can fix with one setting. The model knows only two things: what it learned in training, and what is in the request. Give it the facts and it uses them:

>>> faq = "Reach the boarding point 15 minutes before departure."
>>> question = "How early should I reach the boarding point?"

Asked alone, the lab model says Please reach the boarding point 30 minutes before departure., which is invented. Sent with faq in front of the question, it says Reach the boarding point 15 minutes before departure. Task 2 does this properly, with the whole policy in a system message: the instructions and background that come before the conversation.

Seeing the probabilities

You can ask the API to show the probabilities it picked from. With logprobs=True and top_logprobs=5, each token of the reply comes with its five most likely alternatives. They are log probabilities, the natural logarithm of each probability (always 0 or below), because adding logs is safer than multiplying many tiny numbers. math.exp turns one back:

>>> import math
>>> math.exp(-0.7808)
0.45804
>>> f"{0.45804:.1%}"
'45.8%'

For "How early should I reach the boarding point?", the lab model's first token candidates were Please 45.8%, Sure 25.1%, Happy 15.0%, Of 7.6% and Good 4.2%. Task 3 prints them for another question.

When the API says no

A bad request does not crash on your side: the server refuses it, and the openai package raises an exception named after the HTTP status, carrying the API's own message:

openai.NotFoundError: Error code: 404 - {'error': {'message': 'The model
`lab-chatt` does not exist or you do not have access to it.', 'type':
'invalid_request_error', 'param': None, 'code': 'model_not_found'}}
openai.BadRequestError: Error code: 400 - {'error': {'message': "Invalid
value: 'customer'. Supported values are: 'system', 'assistant', 'user',
'tool', and 'developer'.", 'type': 'invalid_request_error', 'param':
'messages[0].role', 'code': 'invalid_value'}}

Read them like a traceback, from the bottom: the message says what was wrong and param says where. lab models in the terminal lists the lab's model names; lab requests shows exactly what your code sent.

Your turn

Ask the model task 1's question and compare its answer with section 6 of the policy, then make it answer from the policy in task 2. Task 3's trap is the log probability, and the interview question hands you three mistakes, one error at a time.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    first_call.py: ask the model

    Zoya wants to see the model answer a real customer question. Send lab-chat one user message, exactly:

    How much luggage can I take on a SafarLine bus?

    and print only the text of its reply, nothing else.

    Then compare the reply with section 6 of /workspace/data/policy.md: the model has never read it.

  2. Task 2

    grounded.py: answer from the policy

    Lata needs answers from SafarLine's own rules. Read /workspace/data/policy.md, and send lab-chat two messages:

    a system message with an instruction to answer only from the policy, followed by the whole policy text;

    then the same user question as task 1.

    Print only the reply. It should now give the real luggage allowance from the policy.

  3. Task 3

    next_token.py: the model's first choices

    Nikhil wants to see the prediction itself. Ask lab-chat, as one user message:

    Can I change my travel date?

    asking for only one token back, with log probabilities and the 5 most likely first tokens.

    Print one line per candidate, in the order the API gives them, as the token, a colon, a space and its probability as a percentage with one decimal, for example:

    Hello: 12.5%

  4. Task 4 · interview question

    Interview: fix_me.py, read the API's errors

    fix_me.py should ask lab-chat whether there is a bus from Pune to Goa tonight and print the reply, but it has three mistakes. Run it, read each error, fix the line it points at, and run it again until it prints the model's reply.

    Keep the question exactly as it is.

What you will learn

  • What an LLM does: predict the next token, one at a time.
  • Your first call with the openai package, and the response object.
  • Why answers vary and why models invent: sampling, and knowing only what was in training and in the request.
  • Reading the API's errors, the way you read a traceback.

More in Talking to a model

Build "What an LLM is" yourself

SafarLine, an intercity bus company, drowns in WhatsApp messages. You build its first AI assistant. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.