Build an AI gateway (TypeScript) · Chapter 12 of 21

The semantic cache

By the end of this chapter you can explain how a semantic cache decides that two questions mean the same thing, turn text into embeddings and compare them, build a cache that never gives one brand's answer to another, keep it from slowing anything down when the embedding provider is down, and choose its threshold by measuring instead of guessing. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Spend less
  • 4 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Build an AI gateway (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Kaapi Kottai's FAQ widget gets the same few questions all day. "Can I pause my subscription?" comes in at 10:12, and "How do I stop my subscription for a while?" at 15:40. The answer is the same, word for word, and Darwaza pays a provider for it twice.

The exact cache from chapter 11 cannot help. Its key is a hash of the exact words, so "How do I stop my subscription for a while?" and "Can I pause my subscription?" are two different keys. Even "is the fish pickle made with coconut oil" in lower case, without its question mark, misses. Of the day's 121 FAQs, the exact cache answers 61; 45 more are the same questions in other words, and every one of them goes to a provider.

Lakshmi would like those rupees back. Anjali has one condition: a cached answer must never be the wrong answer. A customer who asks whether the fish pickle is made with coconut oil must never get the mango pickle's answer. That condition is most of this chapter.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    similar.ts: how close are two questions?

    Embed three of Thenga Naturals' questions in one request to nimbus-embed (POST ${PROVIDERS.nimbus.baseUrl}/v1/embeddings, body { model: 'nimbus-embed', input: [A, B, C] }): A "Is the coconut oil safe for babies?", B "Can I use the coconut oil for my baby?", C "Is the coconut oil safe for cooking?". Print, in this order:

    numbers per text: N (the length of one embedding)

    paraphrase: X (the cosine of A and B, 3 decimals)

    near miss: X (the cosine of A and C, 3 decimals)

    cost: ₹X for N tokens (usage.prompt_tokens x PRICES['nimbus/nimbus-embed'].in over 1,000,000, 6 decimals)

    The embeddings are unit length, so the cosine is the sum of a[i] x b[i].

  2. Task 2

    semantic.ts: the semantic cache

    Export embed(text): Promise<number[] | undefined> (nimbus-embed's data[0].embedding, with its own 300 ms timeout; undefined on any failure), cosine(a, b) (the sum of a[i] x b[i]), and class SemanticCache: constructor(threshold = 0.85) kept as this.threshold;

    get(req) returns { value?, vec?, score? }: only for a faq at temperature 0 (anything else: {} without embedding); embed the last user message; compare with every answer stored in the same scope (team, feature and system prompt); the best score at or above the threshold is a hit { value, vec, score }, otherwise { vec, score }.

    set(req, vec, value) stores { vec, value } in the request's scope, but not when vec is undefined, value.finish is 'length', or the request is not a temperature-0 FAQ.

  3. Task 3

    gateway.ts: plug it into Darwaza

    Export cache (a new SemanticCache with threshold 0.85) and gateway: createDarwaza({ semantic: cache }) from ./darwaza.ts. Under if (import.meta.main), replay only the day's FAQs through gateway (replay(gateway, { entries: day({ features: ['faq'] }) })), print the report, then:

    semantic hits: N (gateway.stats.semanticHits)

  4. Task 4 · interview question

    Interview: threshold.ts, choose the threshold

    Export chooseThreshold(pairs: { score, same }[]) returning { threshold, caught, missed }: threshold is the lowest value in steps of 0.01 above the highest score of any pair with same false (a different pair at exactly 0.80 gives 0.81); caught counts same pairs with score >= threshold, missed the other same pairs.

    Under if (import.meta.main): for each team, take its distinct FAQ question texts from day({ features: ['faq'] }) (the last message), each with its entry's expect joined by '|'; embed them; pair every two texts of the same team (score: their cosine; same: their expects are equal) and print:

    pairs: N (S same question, D different)

    closest different pair: X (team) (3 decimals)

    threshold: T (2 decimals)

    paraphrases caught: C of S

What you will learn

  • Embeddings: a text as a list of numbers, and what one costs.
  • Cosine similarity: how close two questions are, from 0 to 1.
  • The threshold: how close is close enough, and the near misses that make it hard.
  • The scope: why the cache is split by team, feature and system prompt.
  • What to cache: only FAQs at temperature 0, only whole answers.
  • Failing safe: a cache that never makes an answer slower, even at 14:00.
  • Measuring the threshold on questions whose right answers you know.

More in Spend less

Build The semantic cache yourself

Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build an AI gateway (TypeScript) are free. This one comes with the all labs plan.