Phony.ai

Sub-second voice AI sounds like a model achievement. It isn't. This episode of Phony.ai pulls apart the full end-to-end pipeline that determines how long a real caller actually waits — from the moment they stop speaking to the moment they hear the first syllable of a response — and makes the case that almost every published latency figure is measuring something different, and usually something narrower. The conversation is grounded in the Phony.ai deep-dive on voice response architecture, which traces how the team's production numbers moved from 3.5–4.5 seconds to 2.2–3.2 seconds in a single week — without changing the model.

Here's what the episode covers:

• Why latency comparisons are almost meaningless. Vendors quote response times without defining the measurement window, making cross-vendor benchmarks impossible to interpret with confidence.
• The four-leg pipeline every caller pays for. Endpointing, transcription, model inference, and speech synthesis are sequential — and three of those four legs barely move regardless of how fast the model is.
• The honest floor for endpointing. Pushing the silence threshold below ~300 milliseconds causes the system to clip callers mid-sentence; the clipped fragment transcribes to nothing, and the caller experiences being ignored — a worse outcome than a longer wait.
• Transcription as the current bottleneck. On the web path, transcription accounts for roughly 830 milliseconds and is the single largest contributor — a constraint shaped by the fact that batch transcription can't begin until speech ends.
• The model is the one leg that's actually free to change. A small, fast voice-tuned model comes in around 690 ms to first token; a frontier model runs closer to 2.5 seconds — a factor of ~3.5× on the one variable that isn't constrained by physics or architecture.
• Two architectural moves that actually shift the floor. Speculative execution (generating against a predicted complete utterance and discarding if the caller continues) and speech-to-speech models that collapse the pipeline entirely are the levers that matter — not prompt caching or reasoning-mode toggles, both of which the team investigated and found innocent.
• Geography is a variable, not a footnote. Network distance between a host and a provider can swing results by 150 milliseconds. Any latency figure without a region attached is really a figure about one specific data centre.

The episode closes with a practical challenge: the next time a vendor quotes a sub-second response time, ask exactly what was measured between. Even with a hypothetical zero-latency model, endpointing, transcription, and synthesis still sit in the chain — and they don't disappear with a faster GPU. For more on how Phony.ai approaches product decisions around caller experience, check out the earlier episode Why There's No List Upload — And Why That's the Whole Point.

Phony.ai