How to Choose an Enterprise Voice AI Stack, Not Just a TTS Vendor

Most enterprise voice AI evaluations start and end with a TTS bake-off. A team pulls three or four vendors, runs the same demo phrases through each API, measures time-to-first-audio, and picks whoever clears 200ms. Then it ships, and the agent still feels slow.

Here's why. Time-to-first-audio measures how fast a TTS vendor turns text into sound. It says nothing about how long the user waited before that text existed. The latency a person experiences runs from the moment they stop talking to the moment they hear a response back, and TTS is the last of four stages in that chain, not the whole of it.

The gap shows up on go-live day, not in the vendor demo. A team picks a TTS vendor with a great TTFA number, wires it behind a fast LLM, and the agent still talks over people mid-sentence or leaves two seconds of dead air before it answers. Nobody misconfigured the TTS. The TTS was never where the delay lived. This decision usually comes up while building on top of, or deciding whether to build at all instead of buying, an AI meeting API: interviews, sales calls, support escalations, anything where an agent has to hold a real-time voice conversation with a person.

The latency budget, stage by stage

A realtime voice pipeline has four stages between the user finishing a sentence and hearing a response, plus network overhead between them.

For a whole-pipeline reference point: ORI Utter's published spec is roughly 600-900ms from end of user turn to first audio at P50 (English). That's the number measured the way a user experiences it, across every stage in the table at once, and it's the shape of number to demand from any vendor: end to end, with a percentile attached, not a single-stage benchmark.

The table also explains why stitching four best-of-breed vendors together rarely adds up to a fast stack. Every seam between separately hosted stages costs a network hop, a streaming-format conversion, and a piece of glue code nobody benchmarks, and the latency those seams add belongs to no vendor on your invoice. This is the case for a single audio orchestration platform: ORI Utter runs all four stages, semantic turn detection, streaming STT, a swappable reasoning slot, and the streaming voice head, inside one platform. Owning the handoffs is how the whole pipeline lands in that 600-900ms range, and it makes the end-to-end number one vendor's responsibility instead of nobody's.

Add the stages up and 500ms to a second, start to finish, is the range most teams treat as the line between conversational and sluggish. TTS is one row in that table. Optimize it in isolation and you've addressed roughly 10-20% of the total budget, because the other three stages are still running on whatever was already in place before the bake-off started.

This breakdown is also the shape a procurement conversation should take. Ask each vendor which stage they actually own end to end, request their measured number for that stage specifically, and add the stages up yourself instead of trusting a single blended latency figure on a spec sheet. A vendor selling only TTS can only ever answer for one row of the table above, no matter how the sales deck frames it.

Criterion 1: measure the stack, not the component

The number that matters is end-of-user-turn to first-audio-back, measured across the whole pipeline, not TTS submit-to-first-byte in isolation. Two vendors can both advertise "180ms TTFA" and produce a completely different experience once turn detection, STT finalization, and reasoning are wired in front of them.

Ask for P50 and P90/P99, not just the median. A stack that averages 600ms but spikes to 2.5 seconds on one turn in ten will feel worse than a stack that holds a steady 750ms, because users remember the outlier, not the average.

Test with real multi-turn conversations, not a single scripted utterance read into a quiet room. Interruptions, backchannels ("mhm," "right, go on"), topic changes mid-call, a participant who trails off and restarts. A vendor's demo script is the easiest input any stack will ever see. Production traffic is not.

Criterion 2: turn detection is the hidden latency stage

A fast pipeline with bad turn detection is still a bad product. Fixed-silence VAD, voice activity detection that waits for a set number of milliseconds of quiet before deciding a turn is over, has to pick one number for every user, every sentence, every pause to think. Set it too short and the agent interrupts people mid-thought. Set it too long and every response opens with a beat of dead air, which reads as the system stalling even when the TTS behind it is fast.

Semantic turn detection replaces the fixed timer with a model that decides, based on what was actually said, a trailing "and…" versus a completed sentence, a thinking pause versus a genuine yield, whether the user is done. It's the stage most enterprise buyers skip evaluating entirely, because it doesn't show up on a TTS vendor's spec sheet at all.

ORI Utter's turn engine runs this way: 92%+ correct end-of-turn on our internal meeting evaluation set, the difference between a stack that talks over people and one that waits its turn correctly.

Criterion 3: deployment model and data residency

Enterprise voice deployments come in three shapes. A shared cloud API is fastest to integrate and hardest to defend to a compliance team, because the vendor's infrastructure decisions become your infrastructure decisions by default. A dedicated single-tenant deployment, the same stack running in an environment scoped to just you, removes noisy-neighbor latency variance and gives you audit-friendly logging without owning hardware. On-premises, the full stack running inside your own network, is the answer when the requirement is that voice data never leaves your infrastructure at all: healthcare, finance, and public-sector workloads default to asking for this.

Utter ships both of the latter two: dedicated cloud in a region of your choice, and full on-premises deployment where the turn engine, reasoning layer, and voice head all run inside your network. Both configurations support bringing your own reasoning model and fine-tuning on a customer-scoped checkpoint.

Being direct about where this stands: Utter is in research preview, with a 30-minute session cap and English as the primary supported language today. If an enterprise deployment needs longer sessions or non-English coverage on day one, ask any vendor, including us, for a dated roadmap commitment, not a slide.

Criterion 4: model swappability

The reasoning model you pick today won't be the best one available in eighteen months, and vendors deprecate endpoints on their own schedule, not yours. A voice stack that hardcodes a single LLM behind its audio layer forces a migration project every time that happens. Look for a reasoning layer built as a swappable slot, one where any OpenAI-compatible endpoint can sit behind the audio pipeline without a rewrite.

Utter's reasoning layer is built this way: bring your own agent and point it at any OpenAI-compatible endpoint, or use Waterr's default backbone, and a model deprecation becomes a config change instead of a migration. The audio substrate underneath it, the streaming STT and TTS, runs on open models from Kyutai, the Paris lab behind Moshi, released under a CC BY 4.0 license. That matters less for cost and more for what it lets a security team actually check: the weights are inspectable, which is a real answer when someone asks what's running the speech layer, not just what a sales deck promises.

Criterion 5: multi-turn accuracy under latency constraints

Latency numbers are usually measured on short exchanges: three or four turns, a clean scripted question, a clean scripted answer. Enterprise use cases, support escalations, sales discovery calls, structured interviews, run 20, 30, 40 turns long. A stack that holds sub-second latency at turn 3 and quietly drops a constraint the user stated at turn 5 by the time it reaches turn 30 has a real production problem, and most latency benchmarks won't catch it.

The eval to ask a vendor about is Scale AI's Audio MultiChallenge, which scores multi-turn voice conversations across named failure axes, inference memory, instruction retention, self-coherence, and voice editing, rather than response speed alone. It's a reasonable proxy for whether a stack that's fast in a demo stays coherent across a real session.

Waterr's research here: Monologue, the reasoning harness behind Utter, lifted paired accuracy by +9.59 points on Audio MultiChallenge (95% CI [+2.26, +16.88]) against the same frozen voice backbone, with wall time unchanged. The gain came from letting the model reason in parallel while it listens, not from a slower, heavier response.

Where TTS selection fits

TTS selection still matters: it's one stage in the budget, and picking a poor one breaks the experience. Once the rest of the stack is sound, run this TTS-specific checklist:

• Voice quality and naturalness under your actual domain vocabulary, not a generic demo script
• TTFA at P50 and P90, measured behind your real pipeline, not the vendor's isolated benchmark
• Language and accent coverage against your actual user base
• Pronunciation control for names, acronyms, and domain terms a general model will mangle by default
• Per-minute or per-character cost at your projected call volume, not the list price at low volume

Utter ships 20 production voices as part of the broader stack. The point of listing that here isn't that voice count wins deals. It's that voices are a line item inside a stack decision, not the decision itself, and any evaluation that stops at "which voice sounds best" has skipped the five sections above it. If the voice stack is one piece of a larger AI meeting product you're building or buying, the same math applies one layer up: see the honest bill of materials for building an AI video agent yourself.

The evaluation checklist

A copyable version of the questions above, grouped by what to ask a vendor and the answer that should worry you.

Frequently asked questions

How do I choose a voice AI stack for my enterprise app? Evaluate the whole pipeline, turn detection, STT, reasoning, TTS, as one system, using end-of-user-turn to first-audio at P50 and P90, not a TTS vendor's isolated TTFA number. Then check deployment model against your compliance requirements, model swappability against vendor lock-in risk, and multi-turn accuracy against your actual session length. TTS is the last of five things to check, not the first.

What latency do I need for a realtime voice agent? Most enterprise deployments aim for 500ms to a second, end-of-user-turn to first-audio, to feel conversational. Sub-200ms is a commonly cited target for the TTS stage alone; as a whole-pipeline reference point, ORI Utter's published first-audio spec is roughly 600-900ms at P50 after end of user turn, a range that still reads as natural once turn detection is handled well. Watch P90/P99 as closely as the median; outliers are what users actually notice.

Should I self-host TTS or use an API? Self-host, or run dedicated or on-premises, when data residency or compliance requires voice data to stay inside your infrastructure; healthcare, finance, and public sector usually require this. Use a shared API when speed to integrate matters more than infrastructure control. Either way, TTS deployment is a smaller decision than deployment of the full stack, turn detection and reasoning included, which is what's actually running your session.

Why does my voice agent feel slow even with fast TTS? Because TTS is one of four latency stages, and its TTFA number alone doesn't capture what a user experiences. If turn detection is slow to decide the user is finished, or the reasoning model takes 500ms or more to produce a first token, the pipeline feels sluggish no matter how fast the voice synthesis is downstream. Measure end-of-user-turn to first-audio across the whole stack to find where the time is actually going.

What's the difference between semantic turn detection and VAD? VAD (voice activity detection) decides a turn is over after a fixed period of silence, one number applied to every user and every pause. Semantic turn detection uses what was actually said, a trailing conjunction versus a finished sentence, to decide in real time whether the user is done, which cuts both false interruptions and dead-air delay. It's the stage most latency benchmarks skip measuring entirely.

If you're running this evaluation against your own stack, ORI Utter is Waterr's audio orchestration framework: the same turn engine, reasoning contract, and voice head that run live Waterr meetings today, available standalone for teams bringing their own agent. It's in research preview, honestly capped at 30-minute sessions and English-primary, with dedicated cloud and on-premises deployment for teams that need data residency locked down. The docs have a quickstart if the next step is running these numbers against your own traffic instead of ours.