Skip to content
SignalMD
All posts

Your voice agent feels slow. How to find the slow step.

By Andrew

“Can you check my policy?” the caller asks.

Silence. One second. Three. Five.

“Hello? Are you there?”

The agent comes back with the answer, right on top of the caller. Now both talk at once. The caller sighs and asks for a person.

Your dashboard says the average reply time is 1.6 seconds. That sounds fine. So where did five seconds come from?

This guide shows what callers really wait for and why the average hides it. It shows how to find the slow step on your own calls. Then it shows where SignalMD finds it for you.

What the caller waits for

In normal talk, people answer within a fraction of a second. A long gap on a call feels like a dropped line.

The wait a caller feels runs from the moment they stop talking to the moment the agent starts. Several steps fill it.

What fills the wait after the caller stops talking (invented example) Invented timings for one slow reply. Deciding the caller is done: 0.5 seconds. Turning speech into text: 0.2 seconds. A lookup: 3.5 seconds. Writing the reply: 0.6 seconds. Turning the reply into speech: 0.3 seconds. The lookup is most of the wait. Lookup 3.5 s Caller done? Reply Speech Total: about 5 seconds of silence
An invented example. Each part is small, except one. Fix that one first.
  1. Deciding the caller is done. The agent waits for a short silence to be sure the caller has finished.
  2. Speech to text. The words are written down.
  3. Lookups. The agent calls a tool: a calendar, a policy system, a CRM.
  4. Writing the reply. The model writes its answer. A long prompt or a big model is slower.
  5. Text to speech. The reply becomes a voice.

Why the average hides it

Most replies are fast. A few are very slow. The average mixes them into a number that looks fine.

Reply times on 100 replies (invented example) Invented numbers. 52 replies under 1 second, 33 replies 1 to 2 seconds, 5 replies 2 to 4 seconds, 10 replies over 4 seconds. Every reply over 4 seconds came right after a lookup. The average is 1.6 seconds. 52 33 5 10 Under 1 s 1 to 2 s 2 to 4 s Over 4 s Replies by wait. Average: 1.6 s.
An invented example. The average looks fine. One reply in ten keeps the caller waiting over 4 seconds, and every one follows a lookup.

So look at the slow end, not the middle. Ask: how long did the slowest 1 in 10 replies take? Retell, for example, shows each call’s slow end, and splits the wait into its parts.

But even the slow end does not say where. A slow reply after “hello” and a slow reply after a lookup have different fixes.

Find the slow step by hand

  1. Take 50 calls from the last week. Mix good and bad endings.
  2. List every reply that took over 3 seconds. Your platform’s call record usually has a time for each word, so you can measure the gap.
  3. Write down what came just before. A tool call? A long reply? The first reply of the call?
  4. Group the slow replies by step and tool. One group is usually much bigger than the rest.
  5. Fix that group first. Then count the slow replies again on the new version.

The fix for each cause

Slow replies come afterTry this
A lookupSay a short line first ("Let me check that"). Speed up or cache the tool. Set a time limit with a fallback line.
Any reply, all through the callShorten the prompt. Try a smaller or faster model.
The caller stops talkingShorten the silence the agent waits for. Watch that it does not start cutting callers off.
Long answersMake the first sentence short, so speech starts sooner.
Calls from far awayUse a phone number in the caller's region.
Match the fix to the cause. A faster model does not help a slow calendar.

A shorter wait for silence is a trade. Too short, and the agent talks over callers who pause to think. See callers interrupted when they pause.

Where doing it by hand breaks

Measuring gaps from word times is slow work. Fifty calls can mean a thousand replies.

Tying each slow reply to its step and tool takes the full call record, not only the transcript.

And after the fix, you need the same count on the new version. Otherwise “it feels faster” is the only proof.

Where SignalMD helps

SignalMD tracks a typical reply time across your calls, next to the numbers that matter to callers.

Reply time card: 1.3 seconds typical, with its trend line.
Typical reply time, and how it moved over time.

It also shows each tool’s slow runs, so the slow step has a name.

The Tools panel. For each tool: how often it failed and how often it took 5 seconds or more. The policy lookup never failed but took 5 seconds or more in 9% of 116 runs.
The policy lookup never fails, but about 1 run in 11 takes 5 seconds or more. That is the step to fix.

When callers are left waiting on a tool, it becomes an issue with its place and calls. After you ship a fix, you see whether it worked on real calls. Here is one for a different tool, the availability check.

Did the live version fix it. Callers are left waiting because check availability never answers: from 11% to 4% of calls, fell but not gone.
The wait on "check availability" fell from 11% of calls to 4%. Better, but not gone yet.

Want to find the slow step on your own calls? Try it free or book a call. Short videos are on our YouTube channel.

Common questions

What is good latency for a voice agent?

Aim for replies that feel like normal talk, about a second or less. Then check the slow end, not only the average.

How do I reduce voice agent latency?

Find which step the slow replies follow first. Then fix that step: a lookup, the model, the wait for silence or long answers.

Why does my voice agent feel slow when the average is fine?

A few very slow replies hide inside a good average. Those are the ones callers remember.

See what your agent should fix first.

Book a demo to see the whole loop on a real board, from an issue to the verdict on its fix.

We're letting teams in a few at a time. Join the waitlist and we'll email you when it's your turn.