Build your own GPT project · Chapter 1 of 12
What a language model does
By the end of this chapter you know the one job every GPT does (guess the next token), you have split DabbaDrop's notes into text to learn from and text to test on, you have built the simplest model there is (counting what came next), and you have turned a note into the training examples a GPT really learns from. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Text to numbers
- 3 tasks
- Python
- Free chapter
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.
Why you need this
DabbaDrop cooks home-style lunch dabbas in three Mumbai kitchens: Andheri, Dadar and Thane. Every evening Bhavna in operations writes tomorrow's notes on WhatsApp, in the Hinglish the members read:
Kal Andheri se: Turai Sabzi, Amti, Jeera Rice aur 2 Paratha.
Rider update: Thane ka 12:30 slot thoda late hai, 5 minute mein pahunch jayega.
Meera (Dadar): jain khana mil sakta hai kya? lahsun pyaz nahi. DabbaDrop: Theek hai, kal se ho jayega.
Two years of them are in data/corpus.txt: 1,599 notes.
Farah, the founder, has heard every pitch about AI this year. She does not want another pitch; she wants someone at DabbaDrop who knows what a GPT does inside. Kunal, the senior developer, has a plan: build one from nothing, on these notes, on a laptop, with numpy and nothing else. He calls it ChotaGPT. By chapter 11 it writes notes in Bhavna's style; by chapter 12 it has tests and an API.
Every chapter starts from Kunal's finished code of the chapter before, so the project always runs. You write the chapter's new piece.
Before any maths, one question: what is a language model asked to do?
The lesson
The plan
1 the job: guess the next piece 7 many heads of attention
2 text to numbers: tokenizers 8 a transformer block
3 a first model, and its loss 9 the whole model
4 learning: gradients by hand 10 training, and the long run
5 embeddings and a tiny network 11 writing: sampling, perplexity
6 attention, step by step 12 tests, and the take-home
Tour of the codebase
This is chapter 1, so everything is new. One file is yours.
pyproject.toml the packages and their versions
app/__main__.py python -m app: the API on port 8000, restarted when you save
app/main.py create_app(): the routes
app/config.py where the files are (data/corpus.txt, later the weights)
app/state.py the corpus and its two parts, loaded once
app/routes/corpus.py GET /corpus, GET /next, POST /windows
gpt/ ChotaGPT itself: one file a chapter
gpt/data.py yours: split_text, next_counts, windows, examples_in
data/corpus.txt the 1,599 notes
app/ is the API around the model; gpt/ is the model. There is no login: every saved request just works. Start the API with python -m app and send "The corpus". Before you write anything it answers Internal Server Error [500], and the terminal shows why:
File "/workspace/chapters/w1s/gpt/data.py", line 64, in split_text
raise NotImplementedError("Task 1: split_text is not written yet")
NotImplementedError: Task 1: split_text is not written yet
The one job: the next piece
A GPT reads text and gives, for every possible next piece, a probability. That is all. Writing a whole note is that one guess, repeated: guess the next piece, add it, guess again.
The pieces are called tokens. Chapter 2 builds ChotaGPT's real tokenizer; in this chapter a token is simply one character.
So the question for every model in this project is: after Rider update: Thane ka 12:30 slot thoda l, what comes next? A person who has read Bhavna's notes says a (for "late") without thinking. A model has to learn that from the notes, and nothing else.
Learn on old notes, test on new ones
A model that has read a note can repeat it. That proves nothing: what matters is how well it guesses notes it has never seen. So the notes are cut in two:
- the training text: everything the model may learn from;
- the validation text: notes kept aside, only ever used to measure.
The notes are in date order, so Kunal keeps the newest tenth aside: those are the closest thing to the notes ChotaGPT will be asked to write next. The lines must keep their newline characters, so that the two parts joined give back the corpus exactly. str.splitlines drops them unless you ask:
"chai\nchai\n".splitlines()
"chai\nchai\n".splitlines(keepends=True)
['chai', 'chai']
['chai\n', 'chai\n']
One more trap. int() rounds down, so on a tiny text a tenth can be 0 lines:
int(7 * 0.1), max(1, int(7 * 0.1))
0 1
The finished GET /corpus answers:
{
"characters": 148948,
"lines": 1599,
"distinct_characters": 67,
"train_characters": 134669,
"val_characters": 14279,
...
"first_val_line": "Barish aur chai, saath mein garam pakode."
}
The simplest model: count what came next
Here is a language model with no maths at all. To guess what follows some text, find every place that text appeared before and count what came right after it. The counts are a guess: the more often a character followed, the likelier it is to follow again.
Finding every place takes str.find with a start position. On another string:
s = "roti, roti"
s.find("ot"), s.find("ot", 2), s.find("ot", 8)
1 7 -1
-1 means not found. The trap is where the next search starts. Start it right after the whole context, and you miss places that overlap: "aa" appears three times in "aaaa", at 0, 1 and 2. Start one character after the last place found instead. And a place at the very end of the text has nothing after it: skip it.
Python's Counter does the counting:
Counter("chawal")
Counter({'a': 2, 'c': 1, 'h': 1, 'w': 1, 'l': 1})
The task wants the most common first, and equal counts in character order, so that the answer is the same every time: sort the counter's items with a key, as the task's comment says.
A longer context guesses better
With next_counts written, GET /next asks the notes. Watch the guess sharpen as the context grows (the number is how many times the context appeared, then the top four next characters):
'D' 1190 [('a', 935), ('r', 149), ('e', 70), ('o', 31)]
'Da' 935 [('d', 377), ('l', 337), ('b', 151), ('h', 70)]
'Dal' 337 [(' ', 193), (',', 144)]
'Dal ' 193 [('T', 58), ('F', 56), ('M', 41), ('K', 38)]
'Dal T' 58 [('a', 58)]
After D it could be Dadar, DabbaDrop or Dal. After Dal T there is only Dal Tadka. More context, fewer choices. That is the whole reason a GPT reads a long window of text and not just the last character.
Counting has a fatal limit, though. Ask for a context the notes never contain:
GET /next?context=Pizza
{"error":{"code":"not_found","message":"'Pizza' never appears in the training text"}}
A counting model knows nothing about text it has never seen word for word. A long context is almost never seen word for word. The rest of this project builds a model that generalises: it learns patterns (a menu has dishes, then "aur", then a number of rotis) instead of memorising strings.
Windows: how text becomes examples
A GPT does not learn from one note at a time. It learns from windows: runs of block_size tokens. For each window x there is a target y: the same run moved one place to the right. Position t of y is the token that came after position t of x.
POST /windows shows it on Rider update: Thane with a block of 6:
"windows": 13,
"first_windows": [
{"x": "Rider ", "y": "ider u"},
{"x": "ider u", "y": "der up"},
...
A text of 19 characters gives 13 windows: one for every start where a full y still fits. And one window is not one example. It is six:
"examples_in_first": [
{"context": "R", "next": "i"},
{"context": "Ri", "next": "d"},
{"context": "Rid", "next": "e"},
{"context": "Ride", "next": "r"},
{"context": "Rider", "next": " "},
{"context": "Rider ", "next": "u"}
]
From the first character alone, guess the second; from the first two, guess the third; and so on. Chapter 6 builds a model that reads the whole window once and makes all of those guesses at the same time, each one seeing only what came before it. That is why GPT training is efficient: every window is block_size lessons at once.
Your turn
- Task 1, split_text: keep the newline, keep at least one validation line. The trap:
splitlines()withoutkeepends. - Task 2, next_counts: overlapping places, nothing after the end, the most common first. The trap: searching again from the end of the context.
- Task 3, windows and examples_in:
n - block_sizewindows,yone place right ofx. Kunal will ask: a window of 64 tokens is 64 training examples, not one. Why, and why does that matter for training a GPT?
Start the API with python -m app, send "The corpus", "What comes after Dal Ta?", "What comes after Kal ka?", "Windows of a note" and "A context never seen" (404 not_found), then press Check.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
data.py: keep the newest notes aside
In gpt/data.py write split_text(text, val_fraction=0.1): cut the text into lines that keep their "\n" (splitlines(keepends=True)); n_val is int(number of lines * val_fraction), but at least 1. Return (train, val): the lines before the last n_val joined together, and the last n_val joined. train + val is exactly the text.
The Check splits 25 short lines (val is the last 2), 3 lines (val is the last 1), and 25 lines with val_fraction 0.2, then calls GET /corpus.
- Task 2
data.py: count what comes next
In gpt/data.py write next_counts(text, context): for every place where context appears in text (places may overlap: "aa" appears 3 times in "aaaa"), count the character right after it, skipping a place at the very end. Return a dict {char: count}, the most common first and equal counts in character order: next_counts("dal, daal, dal", "da") is {"l": 2, "a": 1}.
The Check tries that example, "aaaa" with "aa", a context only at the end, and equal counts, then calls GET /next?context=Dal Ta and GET /next?context=Pizza (404).
- Task 3 · interview question
Interview: data.py, windows of examples
In gpt/data.py write windows(seq, block_size): every pair (x, y) where x is seq[i : i + block_size] and y is seq[i + 1 : i + block_size + 1] (the same run moved one place to the right), for i from 0 while a full y fits: a list of n items gives n - block_size pairs. Then examples_in(x, y): for t from 0, the pair (x[:t + 1], y[t]); a window of block_size items holds block_size examples.
The Check tries list("dabba") with block_size 3, range(10) with block_size 4, and examples_in on one window, then calls POST /windows.
Kunal will ask: a window of 64 tokens is 64 training examples, not one. Why, and why does that matter for training a GPT?
What you will learn
- The one job: given the text so far, guess the next piece. Everything a GPT does serves that guess.
- Training text and validation text, and why the newest notes are kept aside.
- A model by counting: what came next, the last times the same text appeared.
- Why a longer context guesses better, measured on the notes.
- Windows: how a text becomes thousands of training examples, and why one window holds many.
More in Text to numbers
Build "What a language model does" yourself
DabbaDrop wants its team to know what a GPT really does inside. You build ChotaGPT, piece by piece, and teach it to write the kitchen's Hinglish notes. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.