Streaming and perceived latency
Mechanism: first signal → useful partial → validated final state
ttft, useful, final = .3, 1.1, 4.8
print("time_to_useful", useful, "time_to_final", final)
Run with python3; expected output separates perceived responsiveness from completion. Stream only content that can safely be shown; buffer strict schemas and high-impact actions until validation passes.
Sources
- Google SRE Book: Addressing Cascading Failures — latency and failure-management context.
- NIST AI RMF — user-facing risk management.
Streaming sends partial output as the model generates it. It does not reduce total generation time, but it reduces perceived waiting and gives the user early evidence that the system is working.
When streaming helps
- Long-form drafting where the user can read while generation continues.
- Conversational answers where progressive text feels natural.
- Coding or writing tasks where partial output can be interrupted.
- Agent runs where progress events can explain what is happening.
Streaming is less useful when the output must be validated as a whole, such as strict JSON, classification, or hidden tool-selection steps.
Design the stream
- Start with a fast status state before tokens arrive.
- Stream meaningful progress, not only raw text.
- Allow cancel, pause, and edit when generation is long.
- Do not show unvalidated structured output as if it were final.
- Mark finality clearly once post-processing and safety checks pass.
Latency budget
Perceived latency has several moments: time to first response, time to useful partial output, time to final answer, and time to user action. Optimize the moment the user actually feels.
Pitfall
Streaming confident hallucinations faster is still a bad product. Pair streaming with grounding, validation, and recovery controls.
Connects to: serving and inference · decoding · latency tradeoffs