Skip to content
SignalMD
All posts

Voice agent test coverage: change one thing, break another

By Andrew

You change one thing in your voice agent, and something else breaks. A prompt edit for one step changes an answer in another. Your tests still pass, because none of them went near the path that broke. This is the hard part of voice agent test coverage.

Older software had a fixed screen. Buttons and forms limited what a person could enter, so a fixed set of tests could reach every path. A voice agent has no such limits. Callers can say anything, in any order. And the model under it is not deterministic: the same words can get a different answer on the next call. So you cannot test every path, and every change can break a path you did not touch.

A common case: every test books, moves or cancels an appointment, because that is what the team built first. Real callers also ask what their plan covers, or when the office is open. No test asks either question. A change that breaks those answers passes every test.

Why voice agent test coverage is a choice

Every test costs something. A test call takes minutes and costs money per minute. A chat test is cheaper, but still takes time to write, run and read. So choose the paths your callers take most. This guide shows how without any tool first, then where SignalMD helps.

Start with a small golden set

A golden set is a short, fixed list of test chats and test calls. Each one says what the caller wants and what must happen. “The caller asks for Tuesday and gets a booking on Tuesday.”

Keep it small. Take each test from a real call, so it sounds like your callers, not like you.

Why it works: the same tests run on every version, so they show what a change broke.

Where it fails: callers change and the set does not. In six months it tests last spring’s callers.

Decide what must be covered

Pick tests in this order. Stop when the budget runs out.

What to cover, in order Five layers, top to bottom. 1, what callers call about: one test per use case with real calls. 2, kinds of caller: the ones who call most in each use case. 3, steps, tools and rules: each one callers reach, at least once. 4, past problems: one test for each problem you fixed. 5, voice situations: talking over, long silence, transfers, which only a test call can play. 1. What callers call about One test per use case that has real calls 2. Kinds of caller The callers who call most, in each use case 3. Steps, tools and rules Each one callers reach, at least once 4. Past problems One test for each problem you fixed 5. Voice situations Talking over, silence, transfers: test calls only
Fill the top layer first. The dashed layer needs a real phone line.
  • What callers call about. Count a month of calls by reason. Every reason with real volume gets a test.
  • Kinds of caller. A first-time caller and a caller with two cars take different paths. Test the common ones.
  • Steps, tools and rules. Every step and tool that callers reach gets at least one test. So does every rule in your prompt, like “confirm the phone number”.
  • Past problems. Each fixed problem gets a test made from the call that showed it. A later change cannot bring it back unseen.

Why it works: the order follows your callers, so the first tests guard the paths most calls take.

Where it fails: a rare call can still hurt. Test a rare call only when a mistake there costs a lot.

Chat tests and test calls do different jobs

A chat test types to your agent. A test call phones it.

What chat tests and test calls can check A table with five lanes. Hearing: chat test no, test call partly. Understanding: both yes. Flow and state: both yes. Speaking: chat test no, test call yes. Integrations: both yes. Below it, chat tests are cheap and fast, and test calls cost more, take longer and fail at random more often. Lane Chat test Test call Hearing No Partly Understanding Yes Yes Flow and state Yes Yes Speaking No Yes Integrations Yes Yes Cost and time Low High
Chat tests check most of the agent for little. Only a call checks how it sounds.

Put most of the golden set in chat. Keep a few test calls for what only a voice can show: talking over, silence, transfers.

Where it fails: a test call uses a test voice in a quiet room. Your callers phone from cars. A test call is “partly” for hearing, never “yes”.

Add a test with every change

Each change can break a path, so the golden set changes with the agent.

  • A new step, tool or rule ships. Add one test that reaches it, before release.
  • A problem is fixed. Add a test made from the call that showed it.
  • A use case goes quiet. If nobody calls about it for weeks, retire its test.
  • A test fails at random. The same words can get a different answer, so run it again once. If it still flips, fix the test or remove it. A flaky test teaches people to ignore red.

Why it works: cost stays flat, because old tests leave as new ones come in.

Passing tests are not proof

A green run says the tests passed this time. The next run can answer differently, and real callers bring words no test has. Read a sample of live calls after each release. When a new kind of call shows up, it becomes a test.

See your gaps on one grid

By hand, this takes two spreadsheets: calls and tests. SignalMD lays them on one grid, on the Tests page, under Coverage.

Use cases run across. Kinds of caller, open issues, steps, tools and rules run down.

The coverage grid: 6 tests, 2 failing, 31 gaps. Use cases run across with their calls and tests. The Ask about coverage column, 54 calls and 0 tests, is marked. Red cells say 1 of 2 failing, and dashed cells say Gap.
The marked column has 54 calls and no test. Dashed cells are gaps, red cells hold failing tests.

A dashed cell is a gap: real callers go there, and no test does. A red cell has a test, but it fails on the live version.

Open a gap to see the calls behind it, and make a test from one in one click.

One gap opened: Checking coverage first, in the Ask about coverage use case. 54 calls in 30 days, no test, a link to open the 50 newest of 54 calls, and a Make a test from a call button. The calls and the button are marked.
Each gap leads to real calls, and a test made from one of them.

The set runs on every version, so a change that breaks a covered path shows up red. A failing test made from a problem names its lane.

Results of the run of all 6 tests on version 8: two booking tests failed, both in the Integrations lane, and are marked. The date of birth test and the tests for booking, moving and cancelling passed.
Two tests fail on this version, both in the Integrations lane.

Want to see this on your own calls? Try it free.

Plan the missing tests in the chat

You can also ask the copilot to plan tests from recent calls, from the agent’s steps and tools, or from your own words. It shows coverage now and after the new tests, marked as planned. Nothing is saved until you press Accept. Accept saves the tests and runs them as chats.

A Tests ready to run card in the copilot, tagged Coverage gaps: 13 cases, coverage of calls 56.5% to 82%, planned, which is marked, with Accept, Decline and Talk about it buttons.
13 cases planned for an 80% target. Coverage counts only after they run.

Common questions

Why does a change in one place break another?

The steps share one model, and often one prompt. A change to one step can change answers in others. Callers also reach steps by paths you did not plan.

How many test calls does a voice agent need?

Start with one test for each reason people call. Add one for each step and tool callers reach, and one for each fixed problem.

Can chat tests replace test calls?

No. Chat tests check what the agent understands, says and does. They cannot check hearing or how the agent sounds. Keep a few test calls for that.

Every change can break a path you did not touch. Your callers show you which paths matter. Want to see where your tests miss them? Try it free or book a call.

See what your agent should fix first.

Book a demo to see the whole loop on a real board, from an issue to the verdict on its fix.

We're letting teams in a few at a time. Join the waitlist and we'll email you when it's your turn.