AI support copilot project · Chapter 12 of 12
Cost, speed and the take-home
By the end of this chapter you can count a request's tokens before you send it and price it in rupees after, keep embeddings and answers in Postgres caches keyed by what they were made from, give each conversation a token budget, and add to a counter in a way two requests at once cannot break. Then the whole copilot is yours, to run with a real model. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Ready for members
- 3 tasks
- Python
- In the all labs plan
The first 2 chapters of AI support copilot project are free. This one comes with the all labs plan.
Why you need this
Rafiq, who looks after DabbaDrop's accounts, asks the question every AI feature meets in the end: what does an answer cost? Today nobody can say. The model reports tokens, and the app returns them, but nothing turns them into rupees. One answer to "How much is the delivery charge?" read 565 tokens and wrote 37.
And the copilot pays for the same work again and again. "How much is the delivery charge?" is one of Sana's most common questions, and every time it is embedded, searched, reranked and answered from scratch. POST /ingest embeds all 162 chunks again when Bhavna fixes one comma. And a member who keeps a conversation going has no limit at all: every turn costs another answer.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
tokens.py: tokens before, rupees after
In app/cost/tokens.py write message_tokens(messages): 3 for every message plus its content's tokens (count_tokens), plus 3 for the reply's start, the way OpenAI counts a chat request. Then cost_inr(tokens, price_input, price_output): tokens is {"input": n, "output": m}, the prices are rupees per million tokens; return (n * price_input + m * price_output) / 1,000,000 rounded to 4 places.
The Check counts a system and a user message and wants the same number the lab model reports as prompt_tokens, then counts an empty list (3), then prices three usages, among them 410 in and 43 out at ₹200 and ₹800 a million (₹0.1164).
- Task 2
cache.py: embed a text once, ever
In app/cost/cache.py write cache_key(text): the sha256 hex of f"{settings.embed_model}\n{text}" (UTF-8). Then cached_embed(db, ai, texts): one embedding per text, in order. Look every key up in one query on EmbeddingCache; send the texts not found (each once, in order) to the model in one embed_texts call, only when there are any; store the new ones with insert(EmbeddingCache).values([...]).on_conflict_do_nothing(index_elements=["key"]) and commit; return the vectors in the texts' order (list(row.embedding) for a stored one).
The Check calls it four times and reads the lab model's request log: [a, b, a] sends 2 texts, [a, b, c] sends 1, [c, a] and [] send nothing; every vector must be its text's, and all three must be in embedding_cache.
- Task 3 · interview question
Interview: budget.py, a token budget per conversation
In app/cost/budget.py write charge(db, conversation_id, tokens): one UPDATE that adds in Postgres, update(Conversation).where(the id).values(tokens_used=Conversation.tokens_used + tokens).returning(Conversation.tokens_used), run with db.scalar; return the new total and do not commit. Then check_budget(db, conversation_id, budget): read tokens_used, and when it is at or over the budget raise AppError(429, "budget_exceeded", "This conversation has used its <budget> tokens: start a new one"); a conversation that does not exist is not checked.
The Check charges 250 and rolls back (nothing committed), then charges 50 while a transaction of its own has charged 100 and holds the row (the total must be 150), then checks a conversation at 2999 and at 3000 tokens, one that does not exist, and POST /conversations/{id}/ask over the budget (429).
Kunal will ask: charge could read tokens_used, add in Python and save it. Why must the adding happen inside the UPDATE?
What you will learn
- Tokens before you send: counting a chat request's tokens exactly as the provider does.
- Rupees after: prices per million tokens, and why four places.
- Caches keyed by content: a hash of what a result was made from, model name included.
- Insert, on conflict do nothing: storing a cache entry that another request may have stored a moment ago.
- A token budget: a limit per conversation, refused with a 429.
- A lost update: what read, add and write does when two requests meet, and the one UPDATE that cannot lose.
More in Ready for members
Build "Cost, speed and the take-home" yourself
DabbaDrop answers the same customer questions all day. You build the copilot that answers them from its own help pages, with sources, and says so when it does not know. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of AI support copilot project are free. This one comes with the all labs plan.