AI engineering traps (TypeScript) · Chapter 2 of 16

Trusting the model's JSON

Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Calling the model
  • 3 tasks
  • TypeScript
  • Free chapter

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.

The lesson

By the end of this case

You can spot model JSON that was trusted without a check, make it happen on purpose with the lab model, check every field before the reply becomes a type, and pin it down with a test that sends replies of the wrong shape.

The ticket

SUN-215: A complaint was "Assigned to undefined". Assigned to you by Farah. Priority: high. Since: this morning.

Rohit, in the operations channel at 9:05 am: "Mr. Srivastava in B-304 had no water since 6 am, and then the lift got stuck on the 3rd floor with his mother in it. He put both in one complaint. The app told him it was assigned to undefined, and no team picked it up for two hours. The guard got her out in the end."

The triage log, from this morning:

06:52:11.904  triage  triage: B-304, water,lift, high
06:52:11.905  tickets complaint 8841 created, team: (none)

Farah: "The model is asked for JSON and it sends JSON, every time. It parsed fine. Look at what came back for a complaint about two things, and at what the code believed it was. Reproduce it first, then fix it, then make sure it can never come back. You are in triage.ts."

JSON is not the same as the right JSON

response_format: { type: 'json_object' } asks the model for JSON, and you get JSON: JSON.parse will not throw. But nothing promises the shape. A model asked for "a category" may send two in a list, a word you never listed, a number as text, or leave a field out. Real models do all of these, especially on inputs nobody tried.

JSON.parse returns any, and an as cast tells the compiler to believe whatever you say. Here the reply for a clubhouse booking has a list where the type says a string, and text where it says a number:

type Booking = { slot: 'morning' | 'evening'; hours: number };

const reply = `{"slot":["morning","evening"],"hours":"2"}`;
const booking = JSON.parse(reply) as Booking;
console.log(`Clubhouse booked: ${booking.slot}, ${booking.hours + 1} hours with cleaning`);
console.log(booking.slot.toUpperCase());

Node prints the first line, then stops at the second (trimmed):

Clubhouse booked: morning,evening, 21 hours with cleaning
file:///workspace/chapters/demo/e.ts:6
console.log(booking.slot.toUpperCase());
                         ^

TypeError: booking.slot.toUpperCase is not a function

typecheck on the same file prints No type errors. The first line is worse than the crash: it is wrong and looks fine.

Checking before you trust

The model's reply is text from outside your program, like a request body. Treat it as unknown and narrow it: check each field the way the compiler understands, and only then build your own type. A list is not a string, and a string you did not list is not one of yours:

const SLOTS: readonly string[] = ['morning', 'evening'];

function slotOf(data: unknown): string | null {
  if (typeof data !== 'object' || data === null || Array.isArray(data)) return null;
  const slot = (data as Record<string, unknown>).slot;
  return typeof slot === 'string' && SLOTS.includes(slot) ? slot : null;
}
evening
null
null

That is slotOf on {"slot":"evening"}, on the list, and on "Evening". When a reply fails the check, the program needs a safe path: here, a person at the Front desk reads the complaint. A library such as zod (in the lab) can do the same checks from a schema.

A test with replies you choose

The lab model answers the ticket's complaint the same way every time, so the repro is the real complaint. The guard is easier with a fake model (case 1): it can send any reply you like, including shapes the real model has not sent yet.

Your turn

Three tasks, one file each. In repro.ts, route the ticket's complaint to undefined on purpose, and run it with node repro.ts. In triage.ts, write parseTriage and use it, so only a reply of the right shape becomes a Triage. In triage.test.ts, write the guard: the check runs it beside the original code, the fix and a wrong fix that takes the first category from a list. Then answer Farah's interview question at the bottom of the case.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    repro.ts: assign the ticket's complaint to undefined

    Write reproduce() in repro.ts. Send the ticket's complaint to route from triage.ts, word for word: Flat B-304 here. No water since 6 am and now the lift is stuck on the 3rd floor too. Resolve to what route gives back.

    The check runs your reproduce() twice: beside the original triage.ts it must resolve to 'Assigned to undefined' (the ticket), and beside a fixed triage.ts to 'Assigned to Front desk'. So it must really route the ticket's complaint. route uses the lab model when you leave out its client; reproduce() must finish within 10 seconds.

  2. Task 2

    triage.ts: check the model's JSON before you trust it

    Write and export parseTriage(text: string): Triage | null in triage.ts. It gives back a Triage only when text is JSON for an object (not a list, not null) whose flat is a string or null, whose category is one of the seven categories as a single string, and whose urgency is 'low', 'medium' or 'high'; for anything else, text that is not JSON included, it gives back null and never throws.

    Then make triage use it, so it resolves to the Triage or to null. route sends a null to the Front desk ('Assigned to Front desk') and logs triage: could not read the model's reply; for a good reply it keeps its log line and team. Keep Triage, Category and TEAMS as they are. The check sends its own replies, broken ones too, and also type-checks triage.ts: parseTriage takes a string and returns Triage | null. Taking the first category from a list is not a fix.

  3. Task 3

    triage.test.ts: a test that would have caught it

    Write a vitest test file, triage.test.ts, that would have caught this bug. The check runs your file three times: beside the original triage.ts, where at least one test must fail; beside the fixed triage.ts, where every test must pass; and beside a wrong fix that takes the first category from a list and checks nothing else, where at least one test must fail.

    So test route with fake models: a good reply goes to its team, and replies of the wrong shape (a list of categories, a category the app does not know, no urgency) go to the Front desk. Use only route, which every version has. Each test must finish within 10 seconds.

What you will learn

  • repro.ts: assign the ticket's complaint to undefined
  • triage.ts: check the model's JSON before you trust it
  • triage.test.ts: a test that would have caught it

More in Calling the model

Build "Trusting the model's JSON" yourself

Sunwai, a Lucknow app for housing societies, runs on a model, and Farah, its CTO, keeps assigning you tickets. Each is a trap AI engineering interviews love to ask about. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.