Email surveys get single-digit response rates and the responses skew to the furious and the delighted. Phone surveys get far better rates and a more representative middle — which is where the useful information lives. They have simply been too expensive to run at any scale.
What changes when a machine asks
Two things, and they pull in opposite directions.
People are blunter. Noticeably so. Telling a machine that the onboarding was confusing is easier than telling the person who did the onboarding. For honest feedback this is a genuine advantage.
People elaborate less. Answers are shorter and less discursive than they would be with a human interviewer. Which means the questions have to do more work.
Designing the questions
Three to five. Not ten. A phone survey is an interruption you were granted, and the goodwill runs out fast. Ask for the three answers you will actually act on.
Open first, scale second. Ask "how did it go?" before "would you rate it out of ten". The reverse anchors them to the number and you lose the reason, which was the valuable part.
One follow-up on the interesting answer. "You mentioned it was slow — where specifically?" This is the whole reason to use a conversational agent rather than a keypad menu, and most surveys never use it.
Say the length up front. "Three questions, about two minutes." Completion rates rise when people know the shape of what they agreed to.
When not to use a conversation
If you only want a number, do not run a conversational agent. "Press 1 to 5 to rate your experience" through a keypad menu costs a fraction, works on any handset, and cannot mishear a 4 as a 5. Reach for a conversation only when the why matters more than the score.
Reading what comes back
Extract the score as a field, not a rating — it is a fact the person stated, and scoring it out of five loses it. Then read the transcripts of everyone who scored in the middle. Detractors tell you what broke; promoters tell you what to say in marketing; the middle tells you what to fix, and it is the group nobody reads.
Timing
Within 48 hours of whatever you are asking about. Later and you get a general impression rather than a memory, and general impressions are not actionable.