Build your own GPT project · Chapter 2 of 12
A tokenizer: characters, then BPE
By the end of this chapter you can turn any note into a list of numbers and back, you have taught a tokenizer ChotaGPT's own vocabulary of 256 tokens from the notes, and you can say why every GPT uses tokens like these instead of single characters or whole words. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Text to numbers
- 3 tasks
- Python
- Free chapter
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.
Why you need this
A model does arithmetic. It multiplies and adds numbers; it cannot multiply the letter K. So before ChotaGPT can learn anything from Bhavna's notes, every note has to become a list of whole numbers, one for each piece of text, and every list the model writes has to become text again. The code that does both is a tokenizer, and each piece is a token.
Chapter 1 used one character as a token. That works, but it is wasteful. Here is one menu line, 54 characters long:
Kal ka dabba, Thane kitchen: Dal Tadka aur Jeera Rice.
In characters it is 54 tokens. ChotaGPT will read windows of 64 tokens, so a window of characters holds barely more than one line. The same line in ChotaGPT's real tokens is 15 tokens, and a window of 64 of those holds about 143 characters. The model sees more of the note for the same work.
Farah asks the plain question: "Why not one number per word, then?" Kunal answers with this chapter: you build both kinds, a character tokenizer and a BPE tokenizer, and measure.
The lesson
Tour of the codebase
New or changed since chapter 1:
gpt/tokenizer.py yours: CharTokenizer, pair_counts, merge, train_bpe,
BpeTokenizer.encode_chunk (the rest is given)
app/routes/tokens.py POST /tokenize, GET /tokenizer
app/state.py now also loads the two tokenizers, once
app/main.py adds the tokens router
scripts/make_tokenizer.py python -m scripts.make_tokenizer: runs train_bpe
data/tokenizer.json ChotaGPT's tokenizer: 67 characters and 189 merges
gpt/data.py is Kunal's finished chapter 1. Start the API with python -m app and send "The same note in characters". Before you write anything it answers Internal Server Error [500], and the terminal says which task is missing:
ids = tok.encode(body.text)
File "/workspace/chapters/w-a-2-start/gpt/tokenizer.py", line 80, in encode
raise NotImplementedError("Task 1: CharTokenizer.encode is not written yet")
NotImplementedError: Task 1: CharTokenizer.encode is not written yet
"Tokenize a note (BPE)" stops at Task 3 instead: the BPE tokenizer is loaded from the file, but it needs encode_chunk to use it.
One id per character
The simplest tokenizer lists every distinct character of the text, sorted, and a character's id is its place in that list. The list is the vocabulary. CharTokenizer.__init__ is given; on a tiny text:
tok = CharTokenizer("chai")
print(tok.chars, tok.stoi)
['?', 'a', 'c', 'h', 'i'] {'?': 0, 'a': 1, 'c': 2, 'h': 3, 'i': 4}
stoi ("string to int") maps a character to its id, and itos maps an id back. Notice the ?: it is always added. It is the unknown token, for a character the tokenizer has never seen. Bhavna writes money as Rs; if a member types ₹, the tokenizer must still give an id, not crash. A dict's get takes a default for exactly this:
prices = {"chai": 15, "samosa": 20}
print(prices.get("chai", 0), prices.get("pizza", 0))
15 0
Decoding is the same lookup the other way, then one join. DabbaDrop's notes use 67 distinct characters (the ? is one of them, members ask questions), so the character tokenizer has 67 tokens. On the finished API, "The same note in characters" answers 54 ids for 54 characters:
{"tokenizer":"char","characters":54,"tokens":54,"ids":[30,42,53,1,52,42,1,45,...],
"pieces":["K","a","l"," ","k","a"," ",...], "decoded":"Kal ka dabba, Thane kitchen: ..."}
Byte pair encoding: glue the most common pair
Byte pair encoding (BPE) starts from those characters and learns bigger tokens from the text itself. The rule is short:
- Count every pair of neighbouring tokens in the text.
- The most common pair becomes one new token, with the next free id.
- Replace that pair everywhere, and go back to 1.
Each new token is a merge. Stop when the vocabulary (characters plus merges) reaches the size you asked for.
Watch it on chai chai chaat. First the text is cut into chunks: a word with the space before it, a number, punctuation, a space. Merges never cross a chunk's edge, so a token never glues the end of one word to the start of the next. chunks is given:
print(chunks("Kal ka dabba (Mon 2 Nov): 3 Roti."))
['Kal', ' ka', ' dabba', ' (', 'Mon', ' 2', ' Nov', '):', ' 3', ' Roti', '.']
The neighbours inside one chunk are what zip gives with the chunk and the chunk moved one place:
print(list(zip("chai", "chai"[1:])))
[('c', 'h'), ('h', 'a'), ('a', 'i')]
In chai chai chaat, c h appears 3 times and so does h a. The finished train_bpe learns five merges up to 12 tokens:
['ch', 'cha', ' cha', 'at', 'chai']
Round 1 is a tie: c h and h a both appear 3 times. Ties are broken by the smaller pair, comparing the two ids (a is 2, c is 3, h is 4: the pair (3, 4) is smaller than (4, 2)). Tuples compare element by element, which is why a key of minus the count, then the pair, works:
print((-3, (3, 4)) < (-3, (4, 2)), (-4, (9, 9)) < (-3, (0, 0)))
True True
Without a fixed rule for ties, two runs on the same text could learn different tokens, and saved weights would mean different things.
Count over chunks, not over the text. The training text has 134,669 characters but only 471 distinct chunks: kitchen alone appears 507 times. Count each distinct chunk once, as a tuple of ids, and add its pairs as many times as the chunk appears. That turns a slow loop over the whole text into a fast loop over 471 chunks.
Merging never overlaps. Replacing the pair a a in a a a gives the new token followed by a: two tokens, not the new token twice. Walk left to right, and after a replacement skip both items of the pair.
The finished code, run with the given script:
$ python -m scripts.make_tokenizer
67 characters + 189 merges = 256 tokens, learned in 0.1 s
the training text: 134,669 characters, 60,390 tokens (2.23 characters a token)
the first merges: 'ha', ' k', 'al', 'ar', 'he', 'ur', ' a', 'at'
wrote data/tokenizer.json
ha comes first because of Thane, chawal, khana and hai. k is kitchen, ka and kal. GET /tokenizer lists the longest tokens learned: DabbaDrop, Andheri, Khichdi, kitchen. The script writes the same data/tokenizer.json that is given, so the rest of the project uses your tokenizer's exact decisions.
Encoding with the merges, in order
Training learned a list of merges in order: rank 0 first, then rank 1, and so on. Encoding a new chunk must make the same decisions in the same order, because later merges are built from the tokens earlier merges made. Start from the characters; each round, look at the neighbouring pairs that have a merge, and apply the one learned first (the lowest rank). Stop when no pair has a merge. Here is Thane with ChotaGPT's tokenizer, one line per round:
rank 0 [' ', 'T', 'ha', 'n', 'e']
rank 22 [' T', 'ha', 'n', 'e']
rank 25 [' T', 'han', 'e']
rank 60 [' T', 'hane']
rank 89 [' Thane']
Picking the leftmost pair instead looks harmless and is wrong: kitchen comes out as ' k', 'it', 'ch', 'en', four tokens where training made one. A word the notes never use still works, it just falls back to smaller pieces:
' kitchen' [111] [' kitchen']
' Paneer' [225] [' Paneer']
' Pizza' [84, 50, 66, 66, 42] [' P', 'i', 'z', 'z', 'a']
The finished "Tokenize a note (BPE)" answers 15 tokens for the 54 characters:
"pieces":["Kal"," ka"," dabba",","," Thane"," kitchen",":"," Dal"," T","ad","ka"," aur"," Jeera"," Rice","."]
Characters, words or BPE
Measured on DabbaDrop's notes:
- Characters: 67 tokens, nothing is ever unknown, but the validation text is 14,279 tokens long and a window of 64 sees one line.
- Whole words: the training text has 571 distinct words (split on spaces), and any word not among them, like
Pizza, would have no id at all. Real English needs hundreds of thousands. - BPE with 256 tokens: the validation text is 6,498 tokens, common words are one token, and a new word falls back to pieces. With 512 tokens (
--vocab 512) the training text is 42,456 tokens, 3.17 characters each: a bigger vocabulary means shorter texts but more numbers in the model.
GPT-2 made the same trade at a larger size: 50,257 BPE tokens.
Your turn
- Task 1, CharTokenizer.encode and decode: the trap is indexing
self.stoi[ch]directly, which raisesKeyError: 'Q'on a character the tokenizer never saw. - Task 2, pair_counts, merge and train_bpe: the traps are counting a chunk's pairs once instead of as often as it appears, overlapping merges (
merge((1, 1, 1), (1, 1), 9)must be(9, 1)), and breaking a tie towards the larger pair. - Task 3, BpeTokenizer.encode_chunk: the trap is merging the leftmost pair instead of the lowest rank, or stopping after one round. Kunal will ask: why do GPTs use BPE tokens instead of characters, or instead of whole words?
Start the API with python -m app and send "Tokenize a note (BPE)", "The same note in characters", "The tokenizer's merges" and "A tokenizer that does not exist" (422). Run python -m scripts.make_tokenizer and check it prints 189 merges. Then press Check.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
tokenizer.py: one id per character
In gpt/tokenizer.py write CharTokenizer.encode(text): one id per character, from self.stoi, with the id of UNKNOWN ("?") for a character the tokenizer has never seen. Then decode(ids): the characters of the ids (self.itos), joined into one string.
The Check encodes "dal" and "dal Q" with CharTokenizer("dal chawal"), decodes back, then calls POST /tokenize with "tokenizer": "char".
- Task 2
tokenizer.py: learn the merges
In gpt/tokenizer.py write pair_counts(words): words maps a chunk's ids (a tuple) to how often the chunk appears; return how often each pair of neighbours appears, each chunk counted as often as it appears. Then merge(ids, pair, new_id): the tuple with every pair, left to right and never overlapping, replaced by new_id. Then train_bpe(text, vocab_size): start from CharTokenizer(text); words are the counted chunks (chunks(text)) as tuples of char ids; while characters plus merges are fewer than vocab_size, take the most common pair (a tie: the smallest pair), give it the id len(chars) + len(merges), add it to merges and merge it in every word; stop early if no pair is left. Return (chars, merges).
The Check tries pair_counts and merge on small tuples, train_bpe on "aaab aaab" up to 6 tokens, and on 20,000 characters of the notes.
- Task 3 · interview question
Interview: tokenizer.py, encode with the merges
In gpt/tokenizer.py write BpeTokenizer.encode_chunk(chunk): the chunk's characters as ids (the id of UNKNOWN for a character it has never seen), then rounds of merging until none applies: each round, among the neighbouring pairs that have a merge (self.ranks), apply the one learned first (the lowest rank) everywhere, as the id self.base + its rank. Return a list of ids.
The Check encodes with hand-made tokenizers (merges ab then bc on "abc"; aa then aaaa on "aaaaa"; an unknown character), then a note with ChotaGPT's own tokenizer, then calls POST /tokenize.
Kunal will ask: why do GPTs use BPE tokens instead of characters, or instead of whole words?
What you will learn
- Text to numbers: what a vocabulary is, and why the model only ever sees ids.
- A character tokenizer: one id per character, and what to do with a character it has never seen.
- Byte pair encoding (BPE): learn new tokens by gluing the most common pair of neighbours, again and again.
- Why ties and order matter: two runs must learn the same tokens, and encoding must repeat training's order.
- Characters, words or BPE: the trade, in DabbaDrop's numbers.
More in Text to numbers
Build "A tokenizer: characters, then BPE" yourself
DabbaDrop wants its team to know what a GPT really does inside. You build ChotaGPT, piece by piece, and teach it to write the kitchen's Hinglish notes. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.