Build your own GPT project · Chapter 9 of 12

The whole model

By the end of this chapter ChotaGPT is one function: token ids go in, a score for every one of the 256 tokens comes out at every position. You add the embeddings that turn ids into numbers (with their place in the window), stack the blocks of chapter 8, score the next token with the same matrix that embedded it, and count every number the model has. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: The whole GPT
  • 3 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Build your own GPT project are free. This one comes with the all labs plan.

Why you need this

Chapter 8's block takes rows of numbers and gives back rows of numbers. A language model has to start from token ids (chapter 2's BPE gives DabbaDrop's notes 256 tokens) and end with a probability for each of the 256 tokens at every position. Three pieces are missing:

  • a way to turn an id like 77 into a row of 64 numbers;
  • a way for the model to know where a token sits in the window;
  • a way to turn the last rows of numbers back into 256 scores.

Farah asks the plain question: "Once it's put together, does it write notes?" Not yet. With fresh random weights the whole model is exactly as good as guessing, and this chapter measures that: a loss of 5.5548, against ln 256 = 5.5452 for picking a token at random. Chapter 10 trains it. But you need the whole model first, and you need to know its size: 169,792 numbers.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    model.py: token and position embeddings

    In gpt/model.py write embed(params, idx, config): idx holds token ids, (B, T) or (T,). If T (the last axis) is more than config.block_size, raise a ValueError. Otherwise each token's row of params["tok_emb"] plus its position's row of params["pos_emb"] (positions 0 to T - 1), shaped like idx plus C.

    The Check embeds (2, 3) and (2,) ids with a small model and asks for 7 tokens with block_size 6.

  2. Task 2

    model.py: the forward pass

    In gpt/model.py write forward(params, idx, config): x = embed(...), then every block in order (block_forward with block_params(params, i) and config.n_head), keeping each block's cache in a list; then layer_norm with lnf_g, lnf_b gives h; logits = h times params["tok_emb"] transposed (weight tying). Return (logits, {"idx": idx, "blocks": [caches], "lnf": the last layer norm's cache, "h": h}).

    The Check compares your logits with its own on a 2-layer model, checks the cache, then calls POST /model/next and GET /model/loss (fresh weights score about ln 256).

  3. Task 3 · interview question

    Interview: model.py, count the parameters

    In gpt/model.py write count_params(params): the number of numbers by group, as Python ints: {"total", "embeddings" (tok_emb, pos_emb), "attention" (every wq, wk, wv, wo), "feed_forward" (every fc_w, fc_b, proj_w, proj_b), "layer_norms" (everything else)}, total being the sum of the four.

    The Check counts a small model (vocab 11, block_size 6, C 8, 2 layers), then calls GET /model.

    Kunal will ask: where do most of ChotaGPT's numbers live, and what would an output matrix of its own cost without weight tying?

What you will learn

  • Token embeddings: a table with one row of numbers per token.
  • Position embeddings: one learned row per place in the window, and why attention needs them.
  • block_size: the longest input the model can read, and why a longer one is an error.
  • The forward pass: embed, every block in order, a last layer norm, the scores.
  • Weight tying: one matrix to read tokens and to score them.
  • Counting parameters: where ChotaGPT's 169,792 numbers live.

More in The whole GPT

Build The whole model yourself

DabbaDrop wants its team to know what a GPT really does inside. You build ChotaGPT, piece by piece, and teach it to write the kitchen's Hinglish notes. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build your own GPT project are free. This one comes with the all labs plan.