Build your own GPT project (TypeScript) · Chapter 9 of 12
The whole model
By the end of this chapter ChotaGPT is one function: token ids go in, a score for every one of the 256 tokens comes out at every position. You add the embeddings that turn ids into numbers (with their place in the window), stack the blocks of chapter 8, score the next token with the same matrix that embedded it, and count every number the model has. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: The whole GPT
- 3 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Build your own GPT project (TypeScript) are free. This one comes with the all labs plan.
Why you need this
Chapter 8's block takes rows of numbers and gives back rows of numbers. A language model has to start from token ids (chapter 2's BPE gives Parda's notes 256 tokens) and end with a probability for each of the 256 tokens at every position. Three pieces are missing:
- a way to turn an id like 200 into a row of 48 numbers;
- a way for the model to know where a token sits in the window;
- a way to turn the last rows of numbers back into 256 scores.
Sayali asks the plain question: "Once it's put together, does it write notes?" Not yet. With fresh random weights the whole model is exactly as good as guessing, and this chapter measures that: a loss of 5.5501, against ln 256 = 5.5452 for picking a token at random. Chapter 10 trains it. But you need the whole model first, and you need to know its size: 99,696 numbers.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
model.ts: token and position embeddings
In src/gpt/model.ts write embed(params, idx, config): idx is B rows of T token ids. If T is more than config.blockSize, throw an Error. Otherwise return a new Tensor [B, T, C] (C is config.dModel) whose row for text b, position t is tok_emb's row idx[b][t] plus pos_emb's row t, number by number.
The Check embeds 2 rows of 3 ids and one row of exactly blockSize ids with a small model (blockSize 6, C 8), and asks for 7 tokens.
- Task 2
model.ts: the forward pass
In src/gpt/model.ts write forward(params, idx, config): x = embed(...), then every block in order (blockForward with blockParams(params, i) and config.nHead), keeping each block's cache in an array; then layerNorm with params.lnf_g and params.lnf_b gives h; logits = matmul(h, params.tok_emb.transpose()) (weight tying), [B, T, V]. Return [logits, { idx, blocks, lnf, h }].
The Check compares your logits with its own on a 2-layer model, checks the cache, then calls POST /model/next and GET /model/loss (fresh weights score 5.5501, about ln 256 = 5.5452).
- Task 3 · interview question
Interview: model.ts, count the parameters
In src/gpt/model.ts write countParams(params): how many numbers the model has, by group: { total, embeddings (tok_emb, pos_emb), attention (every wq, wk, wv, wo), feed_forward (every fc_w, fc_b, proj_w, proj_b), layer_norms (everything else) }, total being the sum of the four. A tensor's .size is its count of numbers.
The Check counts a small model (vocab 11, blockSize 6, C 8, 2 layers), then calls GET /model (99,696 for LONG).
Nikhil will ask: where do most of ChotaGPT's numbers live, and what would an output matrix of its own cost without weight tying?
What you will learn
- Token embeddings: a table with one row of numbers per token.
- Position embeddings: one learned row per place in the window, and why attention needs them.
- blockSize: the longest input the model can read, and why a longer one must be an error.
- The forward pass: embed, every block in order, a last layer norm, the scores.
- Weight tying: one matrix to read tokens and to score them.
- Counting parameters: where ChotaGPT's 99,696 numbers live.
More in The whole GPT
Build The whole model yourself
Parda wants its team to know what a GPT really does inside. You build ChotaGPT from scratch and teach it to write Parda's Hinglish show notes. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build your own GPT project (TypeScript) are free. This one comes with the all labs plan.