LangChain and LangGraph · Chapter 5 of 25
Streaming and batching
By the end of this chapter you can show a reply while the model is still writing it with stream, send many requests at once with batch and limit how many are in flight with max_concurrency, run requests side by side with ainvoke and asyncio.gather, and time each way to choose between them. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: LangChain basics
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of LangChain and LangGraph are free. This one comes with the all labs plan.
Why you need this
invoke waits for the whole reply. For one tenant that is fine. Two things at RoomRaja make it hurt:
- Tanvi tested the assistant on her phone and watched three seconds of nothing before a long answer appeared at once. Tenants think it is broken and type again.
- At 9 am on the 5th, when rent is due, Faisal expects hundreds of messages within minutes. Asked one after another, each waits for the one before it.
Every Runnable you have met has two more ways to run, made for exactly these: stream and batch, plus async twins of all three.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
stream_reply.py: a reply as it is written
Tanvi wants tenants to see the reply start at once, not after a pause. Stream lab-chat's reply to: Can my brother stay over on Saturday night?
Print each chunk's text as it arrives, on one line with nothing between the pieces, and end that line when the reply is done. Then print a second line: Chunks: and how many chunks carried text (skip the empty ones).
- Task 2
batch_limited.py: 31 messages, three at a time
Faisal wants all 31 messages in /workspace/data/messages.jsonl labelled at once. Build a chain: a prompt (the system message Classify the tenant's message as one of: rent, deposit, notice, maintenance, guest, food, room_change, complaint, praise. Reply with the label only. and the human message {message}), a ChatOpenAI for lab-chat-limited, and a StrOutputParser.
lab-chat-limited answers at most 3 requests at a time and refuses a fourth with a 429 error. Run the chain on all 31 messages with one call to batch, never more than 3 at a time, and print one line per message, in the file's order: its id, a space, then the label.
- Task 3
async_answers.py: five slow answers at the same time
lab-chat-slow takes about 4 seconds per answer. Ask it these five questions, in this order, at the same time with ainvoke and asyncio.gather, and print the five replies' texts, one per line, in the questions' order:
Is lunch included?
What is the notice period?
Can I pay rent in cash?
Can my sister stay over?
Is smoking allowed?
Asked one after another they take 20 seconds; together, about 4.
- Task 4 · interview question
Interview: timing.py, batch, async or one by one
Build a reply chain: a prompt (the system message You are RoomRaja's tenant assistant. and the human message {message}), lab-chat and a StrOutputParser. Take the texts of the first 10 messages in /workspace/data/messages.jsonl.
Run the chain on the 10 three ways, timing each with time.perf_counter: one by one with invoke in a loop, then with batch, then with ainvoke and asyncio.gather. Print three lines, the seconds with two decimals: One by one: 2.95s, Batch: 0.41s, Async: 0.38s (your numbers).
The interviewer will ask: RoomRaja must answer 500 tenants at 9 am when rent is due. Batch, async or threads, and what limit would you set?
What you will learn
- stream: the reply in pieces (chunks) as the model writes it.
- batch: many inputs at once, outputs in order, with max_concurrency.
- ainvoke and asyncio.gather: requests that wait together.
- Timing: time.perf_counter around each way.
More in LangChain basics
Build "Streaming and batching" yourself
RoomRaja, a Pune co-living company, answers tenants on WhatsApp by hand. You build its assistant. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of LangChain and LangGraph are free. This one comes with the all labs plan.