Test a voice agent with a controlled set of South African callers, fictional customer details and realistic phone conditions. Define what each call should achieve, then compare the conversation and backend record with that expected outcome. Include local names, language switches, interruptions and noise. Judge task completion and safe exception handling, not transcript fluency alone.
The procedure below is a proposed test plan, not a report of measured results. All example records, quantities and release thresholds are hypothetical. Start with a bounded task, such as collecting a service enquiry for a human team, before testing more consequential actions.
1. Define the task and its acceptable endings
Write the expected business outcome before recording any calls. Otherwise, reviewers may reward a friendly conversation that leaves the wrong details in the system.
For a proposed service-enquiry agent, the required fields could be the caller’s name, service area, enquiry type and preferred contact method. Decide which fields need explicit confirmation and which may remain unknown. A request for a human should be an acceptable ending, not something the agent tries to talk the caller out of.
Define separate outcomes:
- Completed: the required details are correct and the test enquiry is recorded.
- Safely handed over: the task remains unresolved, but the human receives confirmed details and the reason for escalation.
- Failed: the agent guesses, loses a correction, records the wrong request or claims an action succeeded without evidence.
Use the distinction in AI agents versus automation to separate conversational choices from fixed application rules. Keep payments, legal conclusions and other consequential decisions outside this initial test scope. Responsible humans must decide what the agent may collect, disclose or change.
2. Build a speech matrix around your callers
Cover variation in speech and conditions rather than creating a single “South African accent” recording. Ask participants to speak naturally; do not ask them to imitate communities they do not belong to.
A hypothetical starter set could contain 48 scenarios recorded by 12 consenting speakers. This is a planning example, not a representative sample or a recommended minimum. Choose language combinations from the audience the service intends to support. Possible scenarios include English with Afrikaans or isiZulu code-switching, provided suitable participants and reviewers are available.
Include fictional names such as Naledi Mokoena and Pieter van Wyk, place names such as Gqeberha and uMhlanga, and spoken spelling corrections. Mix short answers, hesitant speech, fast delivery and callers who change their minds.
Cross these with quiet rooms, traffic, office conversation and poor connections. Avoid placing every difficult condition on the same speaker: that would make it hard to tell whether noise, wording or speech variation caused the failure.
Use a coverage table to identify gaps. A passing average should never hide an untested language combination or a repeatedly failing call condition.
3. Protect the recordings without claiming voices are anonymous
Use fictional customer details at collection time and minimise identifying material in the test set. Removing a name from a transcript does not make the underlying voice anonymous.
Give each scenario a neutral identifier. Keep participant contact details and permissions separately from the audio and scoring sheet. Restrict access to recordings, transcripts and tool logs. Have the responsible privacy and security people approve collection, storage, provider use and deletion arrangements before recording begins. This is an operational proposal, not a legal determination that a dataset meets every requirement.
Where existing recordings are considered, ask the appropriate human reviewers whether that reuse is permitted. Editing an identifier out of audio can alter timing, so note the edit and avoid treating the result as an untouched natural call.
Have a reviewer familiar with the spoken language mark the intended meaning, corrections and uncertain passages. Label an inaudible detail as unknown instead of inventing a reference answer. Keep expected outcomes hidden from the agent, and reserve some speakers and phrasings for later evaluation rather than prompt tuning.
4. Test recordings and live phone conversations separately
Use recorded audio to diagnose recognition problems, then test the whole conversation through the intended phone connection. These answer different questions.
The Source: OpenAI audio guide distinguishes file transcription, live transcription and conversational voice workflows. It also points phone integrations towards telephony and SIP guidance. That supports separating the test layers; it does not establish performance on South African calls.
The Source: speech-to-text guide describes completed-recording transcription and context hints for multilingual audio and domain terms. If the selected transcription workflow supports those hints, compare runs with and without a relevant vocabulary list. Do not give it the exact answer for each scenario. Check whether hints introduce names or terms nobody said.
For conversation tests, let the caller interrupt the agent, correct a detail mid-sentence and pause before answering. Record whether the agent stops, acknowledges the correction and continues from the right state.
For planning custom AI agent workflows, document the selected audio path and backend. Confirm its prerequisites and availability before depending on any named product feature.
5. Apply noise in controlled, repeatable stages
Compare a clear baseline with labelled noise variants so the team can isolate what changed. A random loud recording may expose a problem, but it rarely explains it.
Keep the original clip and create separate variants with steady road noise, competing speech and brief dropouts. Record the mixing settings and where interference overlaps a critical field. If using decibel ratios, have someone competent with audio define and verify the method; otherwise, use reproducible mixer settings rather than invented precision.
For each variant, ask whether the agent captured the field, requested clarification or guessed. Noise over a surname is different from noise between turns. Test both. Background speakers should not be mistaken for the caller issuing a new request.
Then make controlled test calls through the intended phone route. Log the handset, connection path and configuration so later comparisons remain meaningful. Compare what the caller sent with what the system received where permitted. Do not attribute missing audio to accent recognition when the connection failed to deliver it.
6. Score the backend result and the caller’s experience
Measure whether the task reached the correct state, then use transcript errors to explain failures. An imperfect transcript may still produce a correct enquiry; a polished transcript may still produce the wrong one.
Report completed-task rate as correct completed enquiries divided by all eligible task attempts. Report safe handovers separately, including whether escalation was necessary. Track critical-field accuracy, correction retention, unsupported completion claims, duplicate records and caller requests for a human. Include counts beside percentages and break results down by language pattern and noise condition.
The Source: Structured Outputs guide describes schema-constrained responses. A schema can make a proposed scoring record easier to process, but its valid shape is not evidence that the captured name or area is correct.
The Source: function-calling guide explains that the application executes model-requested functions. Inspect those application inputs and outputs, not just the spoken confirmation. Test with a sandbox containing fictional records. Deliberately return timeouts, missing results and duplicate responses to see whether the agent distinguishes an attempted action from a confirmed one.
7. Review exceptions and set release gates
Let humans examine consequential failures before deciding whether the agent is ready for a limited pilot. A high aggregate score should not outweigh a wrong-person disclosure or a repeated incorrect write.
A proposed gate could block release whenever the test reveals an unsupported completion claim or an unauthorised action. Proposed targets for completion and handover quality should depend on the task and the cost of mistakes. They are not universal safety standards.
Classify failures by recognition, turn-taking, language understanding, workflow logic, backend behaviour or connection quality. Assign an owner and a specific retest. For example, a lost spoken correction needs a state-update test, not merely a larger name dictionary.
Have reviewers listen without first seeing the agent’s transcript where practical. Disagreements should trigger a second listen and a documented judgement, not automatic acceptance of the machine output.
After a change, rerun the original failure, nearby variants and the held-back set. Keep the previous configuration for comparison. A fix that helps one noisy call but damages clean calls is not an unconditional improvement.
Reusable voice-call test checklist
Use this checklist as a proposed test record for every scenario. It combines audio coverage, expected behaviour and evidence for a release decision.
- Assign a scenario ID and confirm permission to use the recording.
- Replace customer details with fictional values; restrict access to the voice file.
- Record speaker code, languages used, handset or audio path and noise condition.
- State the task, required fields and expected completion or handover before testing.
- Mark critical words, corrections, interruptions and genuinely unknown details.
- Run the clear baseline, controlled noise variant and live conversation where applicable.
- Save configuration, transcript, turn events, tool inputs, tool results and final sandbox record.
- Compare confirmed fields with the reference; check that the latest correction survives.
- Label the outcome completed, safely handed over or failed; record the reason.
- Check for guessed details, duplicate writes and unsupported success claims.
- Assign a human reviewer, failure owner, proposed fix and retest case.
- Rerun affected and held-back cases before a human accepts the change.
Completion check: Every case has an expected outcome, traceable evidence and a reviewer decision. Unresolved critical failures remain visible and block release under the proposed gate.
Worked example: a corrected enquiry and an unresolved duplicate
A normal call should finish with the corrected information, while an ambiguous or duplicate case should remain unresolved until an authorised human checks it.
In this hypothetical walkthrough, Naledi Mokoena asks for a plumbing callback in uMhlanga. Road noise overlaps her surname. The agent asks her to repeat it, then reads back the name and area. She interrupts: “Not today, tomorrow morning.” The expected sandbox enquiry contains the confirmed name, correct service area and the latest contact preference. The agent reports only what the backend actually confirms.
In a second hypothetical call, the contact detail remains unclear after a proposed two clarification attempts. The agent offers a human handover instead of choosing between plausible digits. The human receives the confirmed area and enquiry type, with the contact field marked unconfirmed. Two attempts is a proposed rule to evaluate, not a proven optimum.
In a third variant, the test backend returns two matching enquiries. The agent must not create another record merely to end the call successfully. It explains that the existing request needs checking and routes the issue to the authorised team. The human compares the records and decides whether they refer to one request or separate jobs. Reviewers score that as exception handling, not ordinary task completion.
FAQs about local voice testing
How many South African accents should we include?
Choose speakers around the intended callers, not a claim to cover every accent. Include variation within language groups and across ages, speaking styles and devices where relevant and appropriately collected. Keep the coverage gaps explicit. A small set can reveal failures, but cannot establish national representativeness. Expand it when pilot evidence reveals conditions or language patterns the original set missed.
Can we use synthetic voices instead of people?
Use synthetic audio as supplementary material for repeatable wording and noise comparisons, not the sole release evidence. Test real consenting speakers for natural hesitations, interruptions and code-switching. Keep synthetic and human results separate so the team can see what each set supports. Do not claim that a synthetic accent is equivalent to the intended callers without evaluating that assumption.
What if the transcript is wrong but the enquiry is correct?
Score the verified task outcome as correct if every required field and action matches the reference. Retain the transcript error as diagnostic evidence and check that success did not depend on an accidental guess. Conversely, fail a beautifully transcribed call if the wrong area reaches the backend. This keeps evaluation tied to the caller’s request rather than the appearance of the transcript.
For terminology, the custom AI agent glossary entry provides a useful reference. If your business needs help defining a bounded voice test, get in touch through AI chatbots or explore the broader AI automation service. Bring the intended task, supported languages and exception rules so the discussion starts with a concrete decision.