The first time a team reviews AI call transcripts, they read them like a proofreader. Every misheard word gets circled, and the conclusion is that the system is not accurate enough.
That is the wrong test, and it will make you reject something that was working.
The right question
Not "is every word correct?" but "would I have made the same decision from this?"
A transcript that reads "I have five years experience mostly on note and react, notice period is thirty days" has two errors — "note" for "Node", a missing "of". Any recruiter reads it correctly and shortlists the candidate. The decision survived. That transcript did its job.
Compare with one that reads "I have — years experience with [inaudible], notice period is [inaudible]". No obvious spelling errors at all, and completely useless. The second transcript is the failure, and a word-accuracy score would rank it higher.
What to check, in order
1. Did the agent ask everything?
Scan the agent's turns, not the candidate's. If your prompt lists five things and the transcript shows four questions, the prompt has a gap — usually the agent found a natural place to end and took it. This is the single most common real problem, and it is invisible if you only read the candidate's answers.
2. Are the decision fields present?
Years, notice period, location, compensation — whatever your actual filter is. These are the fields the extraction pulls out and the ones you will sort on. If they are there, in any spelling, the call worked.
3. Did the agent handle the awkward moment well?
Every batch has calls where the candidate was driving, or confused about who was calling, or asked something unexpected. Read those specifically. They are where prompts break, and they are a small enough share that you can read all of them.
4. Did it talk too much?
Look at the length of the agent's turns. If a reply runs to four sentences, the person on the line waited fifteen seconds. That is a prompt fix — "keep every reply to one or two short sentences" — and it improves calls more than any model change.
Where errors actually cluster
Predictably, and usefully: proper nouns. Names, cities, colleges, product names. General speech models substitute a common word they know for an uncommon one they do not.
This is fixable by telling the recogniser what to expect rather than by changing models — load the names and terms you know will occur. And it matters less than it looks, because you already know which candidate you called; you rarely need the transcript to tell you their name.
A practical review routine
For a batch of 400, do not read 400 transcripts. Read:
- Ten at random — for general quality and agent talkativeness.
- Every call that scored in the middle — the top and bottom are usually obvious; the middle is where the rubric is being tested.
- Every call under 60 seconds — these are hang-ups, wrong numbers and voicemail, and they tell you about your list rather than your agent.
That is perhaps forty transcripts and an hour of work, and it will tell you everything about whether the batch was any good.