Voice Agents
A voice agent is a relay of three models: ASR turns your speech into text, an LLM decides what to say, and TTS turns the reply into audio. The hard part is not any single stage — it is the clock. Humans expect a reply within roughly 800ms of finishing a sentence; run the stages naively back-to-back and the silence stretches past two seconds, which conversation reads as broken. Production systems win the budget by streaming: ASR transcribes while you are still talking, the LLM starts on partial text, and TTS speaks the first words before the rest are written. Newer speech-to-speech models fuse the stages into one network, trading pipeline control for speed and tone.
Compare the two pipelines against the 800ms line. Naive stages overshoot badly; streaming plus smaller models sneaks under — note the LLM's first token is the fattest slice of the budget either way.
Check yourself
Why does a voice agent care about time-to-first-token rather than time-to-full-answer?
Go deeper (free): Hugging Face Audio Course ↗