Build your own GPT project (TypeScript) · Chapter 1 of 12

What a language model does

By the end of this chapter you know the one job every GPT does (guess the next token), you have split Parda's notes into text to learn from and text to test on, you have built the simplest model there is (counting what came next), and you have turned a note into the training examples a GPT really learns from. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Text to numbers
  • 3 tasks
  • TypeScript
  • Free chapter

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.

Why you need this

Parda sells tickets for twelve of Pune's own cinemas and stages. Every evening Datta, who runs support and the box office, writes tomorrow's notes on Parda's WhatsApp channel, in the Hinglish the members read:

Kal ka show (Sat 2 Nov), The Attic: Puneri Patya, 8:00 pm. Table Rs 599, Standing Rs 399.
Box office update: Moti Mahal Talkies mein 9:30 pm show ke liye 35 seats bachi hain.
Omkar (Hadapsar): bachche ki umar 2 saal hai, ticket lagega kya? Parda: Bilkul, box office ko bata diya hai.

Two years of them are in data/corpus.txt: 1,801 notes.

Sayali, the founder, has heard every pitch about AI this year. She does not want another pitch; she wants someone at Parda who knows what a GPT does inside. Nikhil, the senior backend engineer, has a plan: build one from nothing, on these notes, on a laptop, in plain TypeScript. No machine learning library, no GPU: arrays of numbers and loops you can read. He calls it ChotaGPT. By chapter 11 it writes notes in Datta's style; by chapter 12 it has tests and an API.

Every chapter starts from Nikhil's finished code of the chapter before, so the project always runs. You write the chapter's new piece.

Before any maths, one question: what is a language model asked to do?

The lesson

The plan

1  the job: guess the next piece       7  many heads of attention
2  text to numbers: tokenizers         8  a transformer block
3  a first model, and its loss         9  the whole model
4  learning: gradients by hand         10 training, and the long run
5  embeddings and a tiny network       11 writing: sampling, perplexity
6  attention, step by step             12 tests, and the take-home

Tour of the codebase

This is chapter 1, so everything is new. One file is yours.

package.json                        the packages, and npm run dev
src/server.ts                       listens on port 3000
src/app.ts                          createApp(): the routes, one router a chapter
src/config.ts                       where the files are (data/corpus.txt, later the weights)
src/state.ts                        the corpus and its two parts, loaded once
src/modules/corpus/corpus.routes.ts GET /corpus, GET /next, POST /windows
src/gpt/                            ChotaGPT itself: one file a chapter
src/gpt/data.ts                     yours: splitText, nextCounts, windows, examplesIn
data/corpus.txt                     the 1,801 notes

src/modules/ is the API around the model; src/gpt/ is the model. There is no login: every saved request just works. Start the API with npm run dev and send "The corpus". Before you write anything it answers 500 {"error":{"code":"internal","message":"Something went wrong on our side"}}, and the terminal shows why:

Error: Task 1: splitText is not written yet
    at splitText (/workspace/chapters/01-what-a-language-model-does/src/gpt/data.ts:60:9)
    at <anonymous> (/workspace/chapters/01-what-a-language-model-does/src/state.ts:20:34)
    ...

The one job: the next piece

A GPT reads text and gives, for every possible next piece, a probability. That is all. Writing a whole note is that one guess, repeated: guess the next piece, add it, guess again.

The pieces are called tokens. Chapter 2 builds ChotaGPT's real tokenizer; in this chapter a token is simply one character.

So the question for every model in this project is: after Box office update: Natya Ghar ka 7:30 pm show house, what comes next? Anyone who has read Datta's notes says f (for "housefull") without thinking. A model has to learn that from the notes, and nothing else.

Learn on old notes, test on new ones

A model that has read a note can repeat it. That proves nothing: what matters is how well it guesses notes it has never seen. So the notes are cut in two:

  • the training text: everything the model may learn from;
  • the validation text: notes kept aside, only ever used to measure.

The notes are in date order, so Nikhil keeps the newest tenth aside: those are the closest thing to the notes ChotaGPT will be asked to write next. The lines must keep their newline characters, so that the two parts joined give back the corpus exactly. split('\n') throws them away, and leaves an empty string after the last one:

> 'chai\nchai\n'.split('\n')
[ 'chai', 'chai', '' ]
> 'chai\nchai\n'.split(/(?<=\n)/)
[ 'chai\n', 'chai\n' ]

The second one splits on a regular expression that matches no character at all: (?<=\n) means "a place just after a newline" (a lookbehind). The cut happens there, and nothing is removed.

One more trap. Math.floor rounds down, so on a tiny text a tenth can be 0 lines:

> Math.floor(7 * 0.1)
0
> Math.max(1, Math.floor(7 * 0.1))
1

To take the last lines of an array, slice takes a negative start: ['a', 'b', 'c', 'd'].slice(-1) is [ 'd' ].

The finished GET /corpus answers:

{
  "characters": 165453,
  "lines": 1801,
  "distinct_characters": 66,
  "train_characters": 149213,
  "val_characters": 16240,
  ...
  "first_val_line": "Kal Rangdeep Studio Theatre mein: The Tiffin Thief, English, 4:00 pm."
}

The simplest model: count what came next

Here is a language model with no maths at all. To guess what follows some text, find every place that text appeared before and count what came right after it. The counts are a guess: the more often a character followed, the likelier it is to follow again.

Finding every place takes indexOf with a start position. On another string:

> const s = 'vada pav, vada pav'
> s.indexOf('ad')
1
> s.indexOf('ad', 2)
11
> s.indexOf('ad', 12)
-1

-1 means not found. The trap is where the next search starts. Start it right after the whole context, and you miss places that overlap: "aa" appears three times in "aaaa", at 0, 1 and 2. Start one character after the last place found instead. And a place at the very end of the text has nothing after it: skip it.

A Map does the counting, one key for each character:

> const m = new Map()
> for (const ch of 'chaat') m.set(ch, (m.get(ch) ?? 0) + 1)
Map(4) { 'c' => 1, 'h' => 1, 'a' => 2, 't' => 1 }
> [...m]
[ [ 'c', 1 ], [ 'h', 1 ], [ 'a', 2 ], [ 't', 1 ] ]

Why not a plain object, { [ch]: count }? Because JavaScript puts keys that look like whole numbers first, in number order, whatever order you added them in:

> ({ s: 1, 9: 1, a: 1, 2: 1 })
{ '2': 1, '9': 1, s: 1, a: 1 }

The notes are full of digits (9:30 pm, Rs 240), and the answer must be in a fixed order: the most common first, equal counts in character order. So nextCounts returns an array of { char, count } objects, which keeps whatever order you sort it into. [...m] gives the entries as pairs; map turns each pair into an object, and sort with a compare function puts them in order (a negative result puts a first).

A longer context guesses better

With nextCounts written, GET /next asks the notes. Watch the guess sharpen as the context grows (how many times the context appeared, then the top four next characters):

"K"         1185 "a" 864, "e" 155, "o" 126, "i" 40
"Ka"         864 "l" 650, "t" 112, " " 62, "a" 40
"Kal"        650 " " 453, "a" 197
"Kala"       197 " " 197
"Kala T"     100 "a" 100

After K it could be Kal, Kala Talkies, Kattaa Cafe, Kedar or Kothrud. After Kala T there is only Kala Talkies. More context, fewer choices. That is the whole reason a GPT reads a long window of text and not just the last character.

Counting has a fatal limit, though. Ask for a context the notes never contain:

GET /next?context=Pizza
{"error":{"code":"not_found","message":"\"Pizza\" never appears in the training text"}}

A counting model knows nothing about text it has never seen letter for letter, and a long context is almost never seen letter for letter. The rest of this project builds a model that generalises: it learns patterns (a show note has a venue, then a title, then a time, then the prices) instead of memorising strings.

Windows: how text becomes examples

A GPT does not learn from one note at a time. It learns from windows: runs of blockSize tokens. For each window x there is a target y: the same run moved one place to the right. Position t of y is the token that came after position t of x.

POST /windows shows it on Box office update with a block of 6:

"windows": 11,
"first_windows": [
  {"x": "Box of", "y": "ox off"},
  {"x": "ox off", "y": "x offi"},
  ...

A text of 17 characters gives 11 windows: one for every start where a full y still fits. And one window is not one example. It is six:

"examples_in_first": [
  {"context": "B", "next": "o"},
  {"context": "Bo", "next": "x"},
  {"context": "Box", "next": " "},
  {"context": "Box ", "next": "o"},
  {"context": "Box o", "next": "f"},
  {"context": "Box of", "next": "f"}
]

From the first character alone, guess the second; from the first two, guess the third; and so on. Chapter 6 builds a model that reads the whole window once and makes all of those guesses at the same time, each one seeing only what came before it. That is why GPT training is efficient: every window is blockSize lessons at once.

windows and examplesIn are generic (<T>): the items are characters in this chapter and token ids from chapter 2 on, and the functions never look inside them. slice(i, j) copies items i up to (not including) j, which is all both of them need.

Your turn

  • Task 1, splitText: keep the newline, keep at least one validation line. The trap: split('\n'), which drops every newline.
  • Task 2, nextCounts: overlapping places, nothing after the end, the most common first. The trap: searching again from the end of the context, or counting into a plain object.
  • Task 3, windows and examplesIn: n - blockSize windows, y one place right of x. Nikhil will ask: a window of 64 tokens is 64 training examples, not one. Why, and why does that matter for training a GPT?

Start the API with npm run dev, send "The corpus", "What comes after Kala T?", "What comes after Kal ka?", "Windows of a note" and "A context never seen" (404 not_found), then press Check.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    data.ts: keep the newest notes aside

    In src/gpt/data.ts write splitText(text, valFraction = 0.1): cut the text into lines that keep their "\n" (text.split(/(?<=\n)/)); nVal is Math.floor(number of lines * valFraction), but at least 1. Return [train, val]: the lines before the last nVal joined together, and the last nVal joined. train + val is exactly the text.

    The Check splits 25 short lines (val is the last 2), 3 lines (val is the last 1), and 25 lines with valFraction 0.2, then calls GET /corpus.

  2. Task 2

    data.ts: count what comes next

    In src/gpt/data.ts write nextCounts(text, context): for every place where context appears in text (places may overlap: "aa" appears 3 times in "aaaa"), count the character right after it, skipping a place at the very end. Return an array of { char, count }, the most common first and equal counts in character order: nextCounts("pav, paav, pav", "pa") is [{ char: "v", count: 2 }, { char: "a", count: 1 }].

    The Check tries that example, "aaaa" with "aa", a context only at the end, equal counts, and digits after the context (s 3 times, 2 twice, 1 once), then calls GET /next?context=Kala T and GET /next?context=Pizza (404).

  3. Task 3 · interview question

    Interview: data.ts, windows of examples

    In src/gpt/data.ts write windows(seq, blockSize): every pair [x, y] where x is seq.slice(i, i + blockSize) and y is seq.slice(i + 1, i + blockSize + 1) (the same run moved one place to the right), for i from 0 while a full y fits: an array of n items gives n - blockSize pairs. Then examplesIn(x, y): for t from 0, the pair [x.slice(0, t + 1), y[t]]; a window of blockSize items holds blockSize examples.

    The Check tries [..."parda"] with blockSize 3, the numbers 0 to 9 with blockSize 4, and examplesIn on one window, then calls POST /windows.

    Nikhil will ask: a window of 64 tokens is 64 training examples, not one. Why, and why does that matter for training a GPT?

What you will learn

  • The one job: given the text so far, guess the next piece. Everything a GPT does serves that guess.
  • Training text and validation text, and why the newest notes are kept aside.
  • A model by counting: what came next, the last times the same text appeared.
  • Why a longer context guesses better, measured on the notes.
  • Windows: how a text becomes thousands of training examples, and why one window holds many.

More in Text to numbers

Build "What a language model does" yourself

Parda wants its team to know what a GPT really does inside. You build ChotaGPT from scratch and teach it to write Parda's Hinglish show notes. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.