Delivering model output token-by-token over HTTP/SSE or websockets—better perceived latency and earlier cancellation, with different back-pressure and logging needs.
Streaming changes UX and ops: users see partial answers sooner, but partial JSON requires parsers that tolerate incomplete payloads unless you buffer. Instrument streams in LLM observability to attribute cost and latency accurately.
Contrast
Batch (non-streaming) calls are simpler for offline jobs and strict structured outputs.