Skip to content
SignalMD
All posts

How many calls do you need to know a voice agent fix worked?

By Andrew

You fix your voice agent on Monday. Last week the problem was on 5 of 40 calls. This week it is on 3 of 40. Did the fix work?

You cannot tell yet. With 40 calls, a move from 5 to 3 happens by chance all the time. Trust that number and you may keep a fix that did nothing. Or undo one that worked.

This guide shows how to know a voice agent fix worked, without any tool: what to count, how many calls you need, and where the count goes wrong. Then it shows where SignalMD does the count for you.

A good day proves nothing

Call results move even when nothing changes. Some days bring harder callers. Some days bring more of one kind of call.

Problem share per day when nothing changes Bars for 14 days. Each day has 50 calls and the problem is on 1 call in 10. The daily share goes 10%, 10%, 16%, 14%, 12%, 8%, 6%, 12%, 8%, 14%, 8%, 10%, 8%, 16%. A dashed line marks 10%, the true rate. 0% 10% 20% 16% 6% 16% True rate Day 1 Day 14 Share of calls with the problem, per day
Nothing changed here: 50 calls a day, a problem on 1 call in 10. The daily share still goes from 6% to 16%.

So a drop after a fix can be noise. The question is not “did the number go down?” It is “did it go down by more than it moves on its own?”

Count before and after, the same way

Here is how, step by step.

  1. Pick one number before you change anything. For example, the share of calls where the agent asks for the date of birth twice.
  2. Write down when the change went live. Use the version number. Calls before it are group A. Calls after it are group B.
  3. Judge every call the same way. Same question, same rule, on both sides. A new way to judge moves the number by itself.
  4. Count only calls that could show the problem. A caller who hangs up at the greeting never reached the date of birth step. Leave those calls out on both sides.
  5. Compare the two shares. Then check that you have enough calls to trust the gap.

Why it works: the agent is the only thing that differs between A and B. You hold everything else the same.

Where it fails: you cannot hold your callers the same. More on that below.

How many calls you need

Two things decide it: how common the problem is, and how big a change you expect. A big drop in a common problem shows fast. A small drop, or a rare problem, takes many calls.

Calls needed on each version to see a change From 40% to 20%: about 80 calls on each version. From 20% to 10%: about 200 calls on each version. From 10% to 5%: about 450 calls on each version. From 5% to 2.5%: about 900 calls on each version. From 20% to 15%: about 900 calls on each version. Problem share, before to after Calls on each version 40% to 20% ~80 20% to 10% ~200 10% to 5% ~450 5% to 2.5% ~900 20% to 15% ~900
A big drop in a common problem shows in about 80 calls. A rare problem, or a small drop, takes about 900.

These are the standard numbers for comparing two shares. With them, you see a real change of that size about 4 times in 5. You see a change that is not there about 1 time in 20.

A rule of thumb: say the problem is on 1 call in 5, and you expect the fix to halve it. Wait for about 200 calls on the new version.

Why it works: you choose the size of the change before you look. A lucky day cannot change your mind.

Where it fails:

  • Rare problems. At 1 call in 100, you need thousands of calls. Make a test from a call that showed the problem, and count live calls over a longer time.
  • Peeking. If you look every hour and stop when it looks good, you will see fixes that did nothing. Pick the number of calls first. Then look.

Compare the same kind of calls

The count expects the same callers on both sides. They are rarely the same.

  • Monday brings different calls than Friday.
  • A billing letter goes out, and half the calls are about bills.
  • A holiday week brings new callers who do not know your steps.

When the mix changes, the number moves. Your fix gets the credit or the blame.

What to do: split the count by why people call. Compare booking calls with booking calls. Use full weeks on both sides, so each weekday is in each.

Where it fails: each split has fewer calls. Split too far, and no part has enough calls to trust. Split on one or two things, not ten.

Check that nothing else got worse

A fix can work and still break something else. One change can break a path you did not touch, as our post on test coverage shows. So count two numbers:

  1. The problem you fixed. Did it go down?
  2. One number for the whole agent. For example, the share of callers who got what they called for. Did it stay the same or go up?

Choose both before the change. If you look at twenty numbers, one of them will move by chance. It will look like a finding.

Lower is not the same as fixed

After enough calls, a fix ends in one of five states. Name them, so nobody reads “lower” as “fixed”.

The five ways a fix can end Fixed: The problem stopped on the new version. Next step: Keep a test for it. Lower, not gone: Less often, but still there. Next step: Read the calls that still show it. No change: The gap is inside the normal swing. Next step: Find another cause. Worse: More often than before. Next step: Roll back. Too soon: Not enough calls yet. Next step: Wait, and do not peek. Result Next step Fixed The problem stopped on the new version Keep a test for it Lower, not gone Less often, but still there Read the calls that still show it No change The gap is inside the normal swing Find another cause Worse More often than before Roll back Too soon Not enough calls yet Wait, and do not peek
Five results, five next steps. "Lower, not gone" is the one teams read as fixed.

“Lower, not gone” is the one teams miss. The problem is less common, so the work looks done. Read the calls that still show it. They often have a second cause.

Let SignalMD do the count

SignalMD does this count on your live calls, for each new version of your agent. You do not count by hand.

The Effect table for version 7 to version 8, 320 to 280 calls, marked Observed improvement. Got it done 63% to 74%, plus 10 points. Handed to a person 7% to 8%, Within noise, which is marked. Calls with a failed tool 16% to 10%, minus 6 points. Reply time 1.9 to 1.3 seconds.
Got it done rose 10 points. Handed to a person moved 1 point, inside its normal swing, so it says Within noise.

Each number says if it moved by more than its normal swing. Until a version has enough calls, it says “Not enough calls” and gives no verdict.

Each problem also gets its own answer by version.

Two By version panels. Left: v8 fixed it, none of its 257 calls over 9 days shows it, against 12 of 296 before. Right, marked: No change on v8 beyond noise, 6 of 257 calls on v8, against 12 of 296 before, over 9 days.
Both problems were on 12 of 296 calls before. One went to none. The other halved, and that is still within noise.

Half as often sounds like a win. With these numbers it is still inside the normal swing, so it does not count as fixed.

A By version panel, marked: Fell on v8 but not gone, 4 of 257 calls on v8, against 38 of 296 before, over 9 days. Bars show 2% on v8 and 13% on v7.
This problem fell from 13% to 2%. The page says lower, not gone.

A problem that is lower but not gone says so. You know to read the calls that still show it.

Want to see this on your own calls? Try it free.

Common questions

How long should I wait after a fix?

Until you have the number of calls you chose before the change. Use full weeks, so each weekday is in the count.

Can test calls tell me a fix worked?

Test calls tell you the change runs. Only live calls tell you what it did for your callers.

What if my agent gets few calls?

Count over a longer time, and expect to see only big changes. Make tests from real calls to guard the paths you fixed.

A fix is a guess until enough calls say it worked. Count the same way, wait for enough calls, and check that nothing else got worse. To see this on your own calls, try it free or book a call. For short walkthroughs, see our YouTube channel.

See what your agent should fix first.

Book a demo to see the whole loop on a real board, from an issue to the verdict on its fix.

We're letting teams in a few at a time. Join the waitlist and we'll email you when it's your turn.