RAG in Python · Chapter 6 of 37

Embedding text

By the end of this chapter you can turn Paisa Setu's text into real embeddings with the embeddings endpoint: one text, all 866 chunks in batches and in order, the right errors retried, several batches at once with asyncio, and the question embedded by the same model as the documents. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Embeddings
  • 5 tasks
  • Python
  • In the all labs plan

The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.

Why you need this

Chapter 5 compared vectors that Ira made by hand. Now Kabir wants all 866 chunks as real vectors, so that "money not reached in bank" can find the settlement articles. A model makes the vectors; your code feeds it.

That is where it goes wrong in production. A loop that mixes up the order stores the refund chunk's vector under the login chunk's id, with no error at all. An empty heading gets a whole batch refused. A rate limit at night stops the ingest job. And when Arjun switches models next quarter, every old vector becomes meaningless.

What you build: 5 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    embed.py: embed_one, embed one text

    Write embed_one(text) in embed.py: send text to client.embeddings.create with the model EMBED_MODEL (lab-embed) and return its vector, the embedding of the first item in the response's data, unchanged.

    embed_one("money not reached in bank") is a list of 384 floats starting 0.0348..., 0.0375..., with length 1 (lab-embed normalises). One request, one text.

  2. Task 2

    embed.py: embed_many, every chunk in batches

    Write embed_many(texts, client) in embed.py: embed all the texts with at most MAX_BATCH (256) per request, one request after the other, using the client parameter (the check passes its own). Return Embedded(vectors=..., tokens=...): vectors[i] is the embedding of texts[i], and tokens is the sum of every response's usage.prompt_tokens.

    The 866 chunk texts take 4 requests, in order: 256, 256, 256, then 98. Each item of a response's data has index, the position of its text in that request: put each embedding at the batch's start plus its index. The check's client returns a batch's items in reverse order, so code that appends them as they come fails.

    The endpoint refuses an empty string with a 400 for the whole request. Before the first request, check every text and raise ValueError("Text 1 is empty") (with the text's position) when one is empty or only spaces: with 300 texts where text 280 is empty, nothing may be sent.

  3. Task 3

    embed.py: with_retry, retry what is worth retrying

    Write with_retry(fn, retries, base_delay) in embed.py: call fn() and return its result. When it raises openai.RateLimitError (429) or openai.InternalServerError (500 and above), sleep with time.sleep and call it again: base_delay seconds before the first retry, then twice base_delay, then four times, doubling each time. After retries retries (so retries plus one calls), raise the last error unchanged.

    Any other error is raised at once, with no retry and no sleep: a 400 (an empty string sent with a client whose max_retries is 0) is tried once, and so is a TypeError from your own code. The check uses lab-chat-flaky, which answers 503, then 429, then the answer: fn is called 3 times.

  4. Task 4

    embed.py: embed_many_async, batches at once

    Write embed_many_async(texts, client, concurrency) in embed.py, an async def: cut the texts into batches of at most MAX_BATCH, send them together with asyncio.gather through the client parameter (an AsyncOpenAI), but never more than concurrency requests at once, with an asyncio.Semaphore. Return the vectors in the order of texts.

    The check embeds 1,732 texts (7 batches) with concurrency 3 through its own client, which takes 50 ms a request and returns items in reverse order: more than one request must be in flight at a time, never more than 3, and each vector must be in its text's place.

  5. Task 5 · interview question

    Interview: embed.py, embed_query with the index's model

    A stored index remembers which model built its vectors. Write embed_query(question, index) in embed.py: embed the question with index.model, not EMBED_MODEL, and return its vector. With IndexInfo(model="lab-embed", dimensions=384) it returns the question's 384 numbers.

    With IndexInfo(model="text-embedding-3-small", dimensions=1536) the request must go to that model, which the lab does not have, so the openai.NotFoundError (404) reaches the caller. With IndexInfo(model="lab-embed", dimensions=1536) the vector's length does not match: raise ValueError naming both sizes, like "The index has 1536 dimensions, but lab-embed returned 384".

    Arjun will ask: when the documents move to a better embedding model, do the questions have to change too, and why?

What you will learn

  • The embeddings endpoint: one or many texts in, one vector each out.
  • Batches and order: the 256 limit, and the index on every result.
  • Errors: empty strings, which errors to retry and which never to.
  • Concurrency: batches sent together with asyncio.gather, held back by a Semaphore.
  • One model for both: why questions and documents must share a model.

More in Embeddings

Build "Embedding text" yourself

Paisa Setu, a payments company, needs its help centre to answer merchants correctly, with sources. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.