Voice agent test coverage: change one thing, break another
By Andrew
You change one thing in your voice agent, and something else breaks. A prompt edit for one step changes an answer in another. Your tests still pass, because none of them went near the path that broke. This is the hard part of voice agent test coverage.
Older software had a fixed screen. Buttons and forms limited what a person could enter, so a fixed set of tests could reach every path. A voice agent has no such limits. Callers can say anything, in any order. And the model under it is not deterministic: the same words can get a different answer on the next call. So you cannot test every path, and every change can break a path you did not touch.
A common case: every test books, moves or cancels an appointment, because that is what the team built first. Real callers also ask what their plan covers, or when the office is open. No test asks either question. A change that breaks those answers passes every test.
Why voice agent test coverage is a choice
Every test costs something. A test call takes minutes and costs money per minute. A chat test is cheaper, but still takes time to write, run and read. So choose the paths your callers take most. This guide shows how without any tool first, then where SignalMD helps.
Start with a small golden set
A golden set is a short, fixed list of test chats and test calls. Each one says what the caller wants and what must happen. “The caller asks for Tuesday and gets a booking on Tuesday.”
Keep it small. Take each test from a real call, so it sounds like your callers, not like you.
Why it works: the same tests run on every version, so they show what a change broke.
Where it fails: callers change and the set does not. In six months it tests last spring’s callers.
Decide what must be covered
Pick tests in this order. Stop when the budget runs out.
- What callers call about. Count a month of calls by reason. Every reason with real volume gets a test.
- Kinds of caller. A first-time caller and a caller with two cars take different paths. Test the common ones.
- Steps, tools and rules. Every step and tool that callers reach gets at least one test. So does every rule in your prompt, like “confirm the phone number”.
- Past problems. Each fixed problem gets a test made from the call that showed it. A later change cannot bring it back unseen.
Why it works: the order follows your callers, so the first tests guard the paths most calls take.
Where it fails: a rare call can still hurt. Test a rare call only when a mistake there costs a lot.
Chat tests and test calls do different jobs
A chat test types to your agent. A test call phones it.
Put most of the golden set in chat. Keep a few test calls for what only a voice can show: talking over, silence, transfers.
Where it fails: a test call uses a test voice in a quiet room. Your callers phone from cars. A test call is “partly” for hearing, never “yes”.
Add a test with every change
Each change can break a path, so the golden set changes with the agent.
- A new step, tool or rule ships. Add one test that reaches it, before release.
- A problem is fixed. Add a test made from the call that showed it.
- A use case goes quiet. If nobody calls about it for weeks, retire its test.
- A test fails at random. The same words can get a different answer, so run it again once. If it still flips, fix the test or remove it. A flaky test teaches people to ignore red.
Why it works: cost stays flat, because old tests leave as new ones come in.
Passing tests are not proof
A green run says the tests passed this time. The next run can answer differently, and real callers bring words no test has. Read a sample of live calls after each release. When a new kind of call shows up, it becomes a test.
See your gaps on one grid
By hand, this takes two spreadsheets: calls and tests. SignalMD lays them on one grid, on the Tests page, under Coverage.
Use cases run across. Kinds of caller, open issues, steps, tools and rules run down.
A dashed cell is a gap: real callers go there, and no test does. A red cell has a test, but it fails on the live version.
Open a gap to see the calls behind it, and make a test from one in one click.
The set runs on every version, so a change that breaks a covered path shows up red. A failing test made from a problem names its lane.
Want to see this on your own calls? Try it free.
Plan the missing tests in the chat
You can also ask the copilot to plan tests from recent calls, from the agent’s steps and tools, or from your own words. It shows coverage now and after the new tests, marked as planned. Nothing is saved until you press Accept. Accept saves the tests and runs them as chats.
Common questions
Why does a change in one place break another?
The steps share one model, and often one prompt. A change to one step can change answers in others. Callers also reach steps by paths you did not plan.
How many test calls does a voice agent need?
Start with one test for each reason people call. Add one for each step and tool callers reach, and one for each fixed problem.
Can chat tests replace test calls?
No. Chat tests check what the agent understands, says and does. They cannot check hearing or how the agent sounds. Keep a few test calls for that.
Every change can break a path you did not touch. Your callers show you which paths matter. Want to see where your tests miss them? Try it free or book a call.