Building with LLMs (TypeScript) · Chapter 1 of 24

What an LLM is

By the end of this chapter you can call a language model from TypeScript, read what comes back, explain why its answers vary and why it sometimes invents things, and fix the errors the API sends you. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Talking to a model
  • 4 tasks
  • TypeScript
  • Free chapter

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.

Why you need this

SafarLine runs buses between seven cities, and its customers write on WhatsApp all day: is there a seat to Goa tonight, how much luggage can I take, where is my refund. Lata and one other agent answer every message by hand, and the queue is longer each evening. Zoya, the founder, wants a model to draft the answers.

Everything you have written in TypeScript so far works on data you can see: you read a file, you compute, you get the same result every time. A model is different. It writes text nobody wrote before, it can give two different answers to one question, and it can state something false with complete confidence. Its types tell you the shape of the reply, never whether the reply is true. Before SafarLine lets it near a customer, you need to know what it is, how to call it, and what it does when it does not know.

The lesson

A model predicts the next token

A large language model (LLM) is a program trained on a huge amount of text to do one thing: given some text, predict what comes next. It does not look things up. It works on tokens, pieces of text of a few characters each (chapter 3 counts them), and it writes an answer one token at a time:

  1. It reads everything it was sent.
  2. It gives every token it knows a probability of coming next.
  3. It picks one (more on how in a moment), adds it to the text, and goes back to step 1, until it decides the answer is finished.

Training is what made those probabilities good: after reading billions of sentences, "The bus leaves at" is very likely followed by a time. That is also the model's limit. It knows what text usually says, not what is true for SafarLine, which it has never seen.

The lab model and the openai package

Models run on a provider's servers, and you reach them over HTTP. The openai npm package is the official TypeScript and JavaScript client for OpenAI's API, and many providers accept the same API. This lab has its own model, lab-chat, running inside your lab with the same API, so you need no account and no internet. Two environment variables point the package at it:

$ echo $OPENAI_BASE_URL
http://127.0.0.1:11500/v1

OPENAI_API_KEY is set too. Point those two at a real provider, with your own key, and the same code talks to a real model. Here is a call, in a file pnr.ts that node pnr.ts runs (Node runs TypeScript files as they are: it strips the types and runs the JavaScript):

import OpenAI from 'openai';

const client = new OpenAI();
const response = await client.chat.completions.create({
  model: 'lab-chat',
  messages: [{ role: 'user', content: 'What does PNR mean?' }],
});
console.log(response.choices[0].message.content);
A PNR, or Passenger Name Record, is the code that identifies a booking.
  • new OpenAI() reads the address and the key from the environment.
  • A request names the model and sends an array of messages. Each message is an object with a role ('user' is the customer speaking) and its content. Chapter 2 is all about roles.
  • create sends an HTTP request, so it returns a promise: await it. The file is an ES module, so await works at the top level.
  • The answer is not the response itself: it is in choices[0].message.content.

The response object

Log the whole response and you see what the API sends back (console.log shortens the nested message to [Object]):

{
  id: 'chatcmpl-labcef988929fbbf580a87b',
  object: 'chat.completion',
  created: 1793209003,
  model: 'lab-chat',
  choices: [
    { index: 0, message: [Object], logprobs: null, finish_reason: 'stop' }
  ],
  usage: {
    prompt_tokens: 15,
    completion_tokens: 12,
    total_tokens: 27,
    ...
  },
  ...
}

and the first choice in full, with console.dir(response.choices[0], { depth: null }):

{
  index: 0,
  message: {
    role: 'assistant',
    content: 'Small pets are welcome on board in a carrier.',
    refusal: null
  },
  logprobs: null,
  finish_reason: 'stop'
}
  • choices is an array, because you can ask for several answers at once (chapter 4). Most of the time there is one, choices[0].
  • message is the model's turn: its role is 'assistant' and its content is the text.
  • finish_reason says why it stopped: 'stop' means it finished; 'length' means it hit a limit you set.
  • usage counts tokens: what you sent (prompt_tokens) and what it wrote (completion_tokens). Providers charge for both.

The package ships types for all of it: response is a ChatCompletion, and your editor knows that message.content is string | null (a reply that is a tool call has no text, chapter 17).

Why answers vary, and why models invent

That answer was to "Can I bring my dog on the bus?". Ask three times:

"Of course! Small pets are welcome on board in a carrier."
"Good question. Small pets are welcome on board in a carrier."
"Small pets are welcome on board in a carrier."

The wording changes because of step 3: the model does not always take the most likely token, it samples from the probabilities, so a less likely first word ("Good", "Of") sometimes wins. Chapter 4 controls this.

The content is worse: SafarLine does not carry pets, except guide dogs (policy, section 6.3). The model did not refuse or say it does not know. It produced what bus companies' rules often say, fluently, because that is what prediction does. This is called a hallucination, and it is not a bug you can fix with one setting. The model knows only two things: what it learned in training, and what is in the request. Give it the facts and it uses them:

const faq = 'Reach the boarding point 15 minutes before departure.';
const question = 'How early should I reach the boarding point?';

Asked alone, the lab model says Please reach the boarding point 30 minutes before departure., which is invented. Sent with faq in front of the question, it says Happy to help. Reach the boarding point 15 minutes before departure. Task 2 does this properly, with the whole policy in a system message: the instructions and background that come before the conversation. The model cannot open your files; your code reads the file and puts its text in the request.

Seeing the probabilities

You can ask the API to show the probabilities it picked from. With logprobs: true and top_logprobs: 5, each token of the reply comes with its five most likely alternatives. They are log probabilities, the natural logarithm of each probability (always 0 or below), because adding logs is safer than multiplying many tiny numbers. Math.exp turns one back:

> Math.exp(-0.7808)
0.45803943314699347
> (0.45804 * 100).toFixed(1) + '%'
'45.8%'

For "How early should I reach the boarding point?", the lab model's first token candidates were Please 45.8%, Sure 25.1%, Happy 15.0%, Of 7.6% and Good 4.2%. They live in response.choices[0].logprobs, and here TypeScript helps you: its type is Choice.Logprobs | null, because the field is empty unless you asked for it. Write response.choices[0].logprobs?.content?.[0] and check the result is there before you use it, and the compiler is satisfied for a good reason. Task 3 prints the candidates for another question.

When the API says no

A bad request does not crash on your side first: the server refuses it, and the openai package throws an error whose class is named after the HTTP status, carrying the API's own message. Unhandled, Node prints it:

NotFoundError: 404 The model `lab-chatt` does not exist or you do not have access to it.
    at APIError.generate (file:///workspace/node_modules/openai/core/error.mjs:50:20)
    ...
BadRequestError: 400 Invalid value: 'customer'. Supported values are: 'system', 'assistant', 'user', 'tool', and 'developer'.

Read the line that names the error, not the stack under it: the class (NotFoundError is 404, BadRequestError is 400) and the API's message say what was wrong. Caught, the error also says where:

try {
  await client.chat.completions.create({ ... });
} catch (err) {
  if (err instanceof OpenAI.APIError) console.log(err.status, err.param, err.code);
}
400 messages[0].role invalid_value

Some mistakes never reach the server. typecheck (the TypeScript compiler, with the lab's settings) knows the package's types, so it rejects a role it does not know and reading .message on the choices array, before anything is sent. A model name is only a string to it: whether lab-chatt exists, only the server can say. lab models in the terminal lists the lab's model names; lab requests shows exactly what your code sent.

Your turn

Ask the model task 1's question and compare its answer with section 6 of the policy, then make it answer from the policy in task 2. Task 3's trap is the log probability, and the interview question hands you three mistakes, one error at a time.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    first-call.ts: ask the model

    Zoya wants to see the model answer a real customer question. Send lab-chat one user message, exactly:

    How much luggage can I take on a SafarLine bus?

    and print only the text of its reply, nothing else.

    Then compare the reply with section 6 of /workspace/data/policy.md: the model has never read it.

  2. Task 2

    grounded.ts: answer from the policy

    Lata needs answers from SafarLine's own rules. Read /workspace/data/policy.md, and send lab-chat two messages:

    a system message with an instruction to answer only from the policy, followed by the whole policy text;

    then the same user question as task 1.

    Print only the reply. It should now give the real luggage allowance from the policy.

  3. Task 3

    next-token.ts: the model's first choices

    Nikhil wants to see the prediction itself. Ask lab-chat, as one user message:

    Can I change my travel date?

    asking for only one token back, with log probabilities and the 5 most likely first tokens.

    Print one line per candidate, in the order the API gives them, as the token, a colon, a space and its probability as a percentage with one decimal, for example:

    Hello: 12.5%

  4. Task 4 · interview question

    Interview: fix-me.ts, read the API's errors

    fix-me.ts should ask lab-chat whether there is a bus from Pune to Goa tonight and print the reply, but it has three mistakes. Run it, read each error, fix the line it points at, and run it again until it prints the model's reply.

    Keep the question exactly as it is.

What you will learn

  • What an LLM does: predict the next token, one at a time.
  • Your first call with the openai package, and the response object.
  • Why answers vary and why models invent: sampling, and knowing only what was in training and in the request.
  • Reading the API's errors, and which of them TypeScript catches first.

More in Talking to a model

Build "What an LLM is" yourself

SafarLine, an intercity bus company, drowns in WhatsApp messages. You build its first AI assistant, in TypeScript. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.