LangChain and LangGraph · Chapter 23 of 25

Testing

By the end of this chapter you can test a chain, a node, a whole graph and an agent's tool calls with a fake chat model that needs no server, keep tests from leaking into each other, and say what only a test with the real model can tell you. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Production
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of LangChain and LangGraph are free. This one comes with the all labs plan.

Why you need this

Tanvi wants one small change to the reply prompt. Sid makes it, and on Monday Mrunal finds that complaints are going to the model instead of to her. Nobody noticed, because nobody tried a complaint after the change.

Tests notice for you: small functions that run your code and check what comes back, every time anything changes. You met pytest in Python; the new problem here is the model. A real model is slow, costs money, needs a server, and words its answers a little differently each time. A test that waits for it and compares its words fails for no reason. This chapter separates your code, tested with a fake model in milliseconds, from the model's choices, tested with the real one.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    test_reply.py: a chain with a fake model

    assistant.py (do not change it) is RoomRaja's assistant in three pieces, each taking the chat model as an argument. Write pytest tests for reply_chain(model) and its steps tenant_inputs and tidy, using GenericFakeChatModel so no request reaches a server. They must pass on assistant.py as it is.

    The check runs them again on three copies with one bug planted in each: the prompt gets the tenant's full name instead of the first name; the reply keeps the spaces the model put around it; an empty reply goes to the tenant instead of the apology. Each copy must make at least one test fail, and your tests must send no request to the lab model.

  2. Task 2

    test_router.py: a node and a whole graph

    Write pytest tests for build_router(model): call route alone with a state for each intent, and run the whole graph with a fake model whose replies you script. They must pass on assistant.py, send no request to the lab model, and fail on each of three copies: a complaint is answered by the model instead of going to Mrunal; the model's word is not read without case and a full stop; every ticket is raised as housekeeping.

  3. Task 3

    test_tools.py: tool calls with a scripted model

    Write pytest tests for build_agent(model): give it a fake model that first asks for a tool (an AIMessage with tool_calls) and then answers in text, and check the ToolMessage the agent sent back and its final answer. They must pass on assistant.py, send no request to the lab model, and fail on each of three copies: the agent is built without the rent_due tool; raise_ticket sends back only the ticket id; ticket_status lets the error for an unknown ticket out.

  4. Task 4 · interview question

    Interview: test_real.py, what the real model chooses

    Write one test that runs build_agent with a ChatOpenAI for lab-chat (temperature=0) on the tenant's message Wi-Fi not working in VMN-203 since last night, and checks that the model's first tool call is raise_ticket with room_id VMN-203 and issue wifi.

    The check runs it on assistant.py (it must pass), on a correct copy whose tool result has one more key so the model's last words differ (it must pass too), and on a copy whose raise_ticket calls its room parameter where (it must fail). The interviewer will ask: what do you test with a fake model, what with a real one, and how often do you run each?

What you will learn

  • Fake chat models: GenericFakeChatModel and scripted replies.
  • Passing the model in: why code that takes the model is testable.
  • Testing a chain, a node and a graph without a server.
  • Testing tool calls: a scripted model that asks for a tool.
  • What only a real model can tell you, and how to test it.

More in Production

Build "Testing" yourself

RoomRaja, a Pune co-living company, answers tenants on WhatsApp by hand. You build its assistant. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of LangChain and LangGraph are free. This one comes with the all labs plan.