Build your own GPT project (TypeScript) · Chapter 6 of 12

Self-attention, step by step

By the end of this chapter you have written the idea at the heart of every GPT: self-attention, where each token looks back at the tokens before it and takes a mix of what they carry, with a mask so that it can never look ahead. You run one head of it on a Parda note and read its weights. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Attention
  • 3 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Build your own GPT project (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Chapter 5's MLP reads three characters and scores 1.5876, far better than the bigram's 2.2969. To read more, the obvious move is to join more rows: 64 tokens of 16 numbers each would be 1,024 numbers going into the first layer. That plan has three problems:

  • the first layer's matrix grows with the context: 1,024 rows of weights where the MLP had 48;
  • every position has its own weights, so "Kala Talkies" at position 5 and at position 40 are learned separately;
  • the context is fixed: a text of 10 tokens and one of 64 need different networks.

Datta's notes need more than three characters. In "Kal ka show (Fri 6 Nov), Kala Talkies: Ghar Wapsi Express, 9:30 pm. Balcony Rs 240", the price depends on the venue, many tokens back. ChotaGPT reads a block of 64 tokens.

Nikhil's answer is attention: the same small set of weights, used at every position, letting each token choose which earlier tokens matter to it.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    attention.ts: the causal mask

    In src/gpt/attention.ts write causalMask(T): T rows of T booleans (boolean[][]), where mask[i][j] is true when position i may look at position j, which is when j <= i (itself and every earlier position), and false above the diagonal.

    The Check tries T = 4 and T = 1, then (once all three tasks run) calls POST /attention, where the first token may only look at itself.

  2. Task 2

    attention.ts: scaled dot-product attention

    In src/gpt/attention.ts write attention(q, k, v), each [..., T, d]: the scores are matmul(q, k.transpose()) times 1 / Math.sqrt(d) (d = q.last). Where causalMask(T) is false the score becomes -Infinity: row r of the scores belongs to position r % T. weights = softmax(scores) (chapter 3's); return [matmul(weights, v), weights].

    The Check compares with its own attention on [5, 4] tensors and on [2, 3, 6, 8] tensors, and looks for a lookahead, rows that do not add up to 1, and missing scaling.

  3. Task 3 · interview question

    Interview: attention.ts, one head of self-attention

    In src/gpt/attention.ts write selfAttention(x, Wq, Wk, Wv): one head. q, k and v are matmul(x, Wq), matmul(x, Wk) and matmul(x, Wv); return what attention(q, k, v) returns.

    The Check runs a [6, 8] x through [8, 4] matrices, then calls POST /attention, where the first token may only look at itself and every row adds up to 1.

    Nikhil will ask: why divide the scores by sqrt(d)? What happens to softmax without it when d is 64?

What you will learn

  • The causal mask: a table of which positions may look at which, and why a GPT must never see the future.
  • Queries, keys and values: what each token asks, offers as a label, and passes on.
  • Dot products as matches, and softmax turning matches into weights.
  • Masking with -Infinity, before softmax and not after.
  • Why the scores are divided by sqrt(d).
  • Self-attention: q, k and v all made from the same tokens.

More in Attention

Build "Self-attention, step by step" yourself

Parda wants its team to know what a GPT really does inside. You build ChotaGPT from scratch and teach it to write Parda's Hinglish show notes. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build your own GPT project (TypeScript) are free. This one comes with the all labs plan.