Key takeaways

An AI voice agent is a system that answers phone calls while holding a natural conversation, queries real systems (CRM, ERP, order platform) and either resolves or routes the request. It differs from an IVR in that there are no menus or numbered options: the caller explains the problem in their own words. The metric that makes it viable is latency under 500 ms.

Why the IVR is out of road

An IVR (Interactive Voice Response) is the numbered phone menu system. Its limit is structural: it forces the caller to translate their problem into an option tree someone designed years ago. When the problem fits no branch, the caller presses zero or gives up.

The numbers reflect it: a classic IVR retains 20-30% of calls; the rest end with a human. And a meaningful share of the ones it does retain arrive misrouted, generating transfers and repeated context: the leading source of dissatisfaction in phone support.

What a voice agent is NOT: it is not an IVR with better speech synthesis. If the system still has an option tree and only swaps the robotic voice for a natural one, you have changed the aesthetics without touching the problem. The real difference is that the agent understands open language and queries systems to resolve, not just to classify.

The analogy: an IVR is a form read aloud. A voice agent is a person at the counter who listens, checks the system and either solves it or takes you to whoever can.

The two architectures: cascade vs speech-to-speech

This is the technical decision that most shapes the outcome, and the one almost no vendor explains properly.

Cascade architecture (STT → LLM → TTS). Voice is transcribed to text (speech-to-text), the text is processed by a language model, and the response is converted back to speech (text-to-speech). Three independent components.

Speech-to-speech architecture. A single multimodal model processes input audio and generates output audio without passing through intermediate text. Fewer hops, less accumulated latency, and prosody and non-verbal cues are preserved.

Criterion Cascade (STT→LLM→TTS) Speech-to-speech
Typical latency Higher (sum of three stages) Lower (single stage)
Naturalness and interruptions Good with careful design Superior
Conversation auditability High (transcript at each step) Lower (needs separate transcription)
Control over what is said High (text can be validated) Lower
Uncommon languages and accents Depends on the STT engine Depends on the model
Fit for regulated sectors Better Needs added controls

Our recommendation: in regulated sectors, or wherever every response must be auditable (finance, healthcare, insurance) cascade remains the more defensible choice, because you have verifiable text at each stage and can validate the response before synthesising it. In high-volume general support, where naturalness matters more than fine-grained auditing, speech-to-speech wins.

And a third option that works well: hybrid, with speech-to-speech for the conversation plus a text validation layer before any action that affects systems.

Latency: the variable that decides everything

In a human phone conversation, the silence between turns runs about 200-300 ms. That is the perceptual bar, and it explains why latency outranks every other technical metric:

Three things that raise latency and are worth watching from the design stage:

  1. Every external system call. Querying the CRM adds its own latency. If the CRM takes 900 ms, no model optimisation compensates.
  2. Network distance. A model served far from the call termination point adds a round trip. Co-locating inference with the voice network is what separates the fastest providers.
  3. Turn design. A system that waits for full silence before processing loses hundreds of milliseconds that incremental processing recovers.

Key market data

Our reading: the jump from 20-30% to 60-80% resolution is real, but only holds if the agent has system access. A voice agent without CRM integration understands the question better and still cannot answer it.

When it makes sense to replace the IVR (and when it doesn't)

Signals it's time:

When it does NOT make sense yet:

Common use cases

Case 1: Replacing the entry menu with conversational triage.

Case 2: Appointment management by phone.

Case 3: Overflow handling at peak times.

How to implement it step by step

  1. Analyse your calls before designing anything. Classify a real sample by intent and measure what share each type represents. Almost always three or four intents concentrate most of the volume.
  2. Start with the most repetitive, most objective intent. Don't try to cover the whole tree on day one. An agent that resolves 40% of calls very well beats one that resolves 100% poorly.
  3. Connect systems before polishing the voice. The agent needs real data. Without a query API the project stalls no matter how good the voice sounds.
  4. Set a latency budget per component. How long recognition, the model, the CRM query and synthesis may each take. If the sum exceeds 800 ms, redesign: don't tune.
  5. Design the transfer as part of the product. With full context to the human agent and no repetition asked of the customer. It has the biggest impact on satisfaction and is usually done worst.
  6. Give a clear exit to a human. "If you'd rather speak to someone, just say so at any point." Hiding the exit does not reduce transfers, it reduces satisfaction.
  7. Listen to real calls every week during the first months. No metric replaces hearing ten full conversations: design flaws surface there before they show on a dashboard.

Common mistakes (and how to avoid them)

Mistake: measuring the project by calls contained. → Reality: containing a call without resolving it just displaces the problem, and the customer calls back angrier. The right metric is first-call resolution, not containment.

Mistake: changing the IVR's voice and calling it an AI agent. → Reality: if there is still an option tree, you still have an IVR. The difference lies in open language and system access, not in speech synthesis.

Mistake: not budgeting integration latency. → Reality: a slow CRM ruins the experience however fast the model is. Latency is designed end to end, including every external query.

Mistake: hiding the option to speak to a person. → Reality: it increases frustration and negative reviews without reducing actual human contact. Offer it explicitly and early.

Mistake: deploying without listening to real calls. → Reality: aggregate metrics do not capture that the agent interrupts, misses an accent or repeats a stock phrase. Ten calls heard in full beat a weekly dashboard.

Mistake: not meeting the duty to disclose it is an AI system. → Reality: since 2 August 2026, Article 50 of the AI Act requires clear disclosure when a user interacts with an AI system. On a call, that means saying so at the start, not in a legal notice.

Realistic timelines and effort

Metrics to track from day 1: first-call resolution rate (not containment), end-to-end p95 latency, transfer rate with reasons, average conversation time versus the human baseline, mid-conversation abandonment rate and CSAT compared to the human channel. Mid-conversation abandonment is the best early indicator that something in the design is off.

Frequently asked questions

What is the difference between an IVR and an AI voice agent?

An IVR presents numbered menus and forces callers to fit their problem into a predefined tree. An AI voice agent understands open language, queries real systems to resolve and transfers with context when it cannot. The difference is comprehension and data access, not the voice.

How much latency can a voice agent have?

Below 500 ms the conversation feels natural; 500-800 ms is perceptible but tolerable; above 800 ms callers start talking over the system and the conversation degrades. Latency is measured end to end, including external system queries.

What is speech-to-speech architecture and how does it differ from cascade?

Cascade transcribes voice to text, processes the text with an LLM and synthesises the reply: three stages, more latency and more auditability. Speech-to-speech uses a single model that processes and generates audio directly: less latency and more naturalness, with less control over what gets said.

How much can a voice agent resolve compared to an IVR?

Classic IVRs retain 20-30% of calls, while AI voice agents reach 60-80% resolution within the first 90 days. That jump only holds if the agent has real access to the systems holding the information.

Do you have to tell callers they are speaking to an AI?

Yes. Since 2 August 2026, Article 50 of the AI Act requires clear disclosure when a person interacts with an AI system. On a call, that means stating it explicitly at the start of the conversation.

Can a voice agent transfer to a human?

It must be able to, and with full context: who is calling, what they asked and what has already been done. Clean transfer is the part of the project with the biggest impact on satisfaction and the most neglected in deployments.

Does it handle accents and multiple languages well?

Current systems handle major languages and accents well, but test with real recordings of your customers before deciding. In cascade architecture quality depends on the recognition engine; in speech-to-speech, on the model.

Which cases should not be automated by voice?

Conversations with high emotional load (serious complaints, sensitive incident notifications, delicate cancellations) and those whose data is not available via API. There the agent's job is to identify the situation and transfer fast, not to resolve.

Does your IVR resolve three calls out of ten?

At Naxia we build voice agents connected to your real systems: conversational triage instead of menus, resolution of your most repetitive intents, context-carrying transfer to your team and a latency budget designed end to end. With the architecture that fits your sector, including the auditable cascade option if you operate in a regulated environment.

If you want to know what share of your current calls could be resolved without human intervention, talk to us. No commitment, no 40-slide decks.

Book a free demo →

Or explore our implementation process first.