This month, we’ve got some thoughts to share on quality control for phone surveys, as well as some insights from our call center in Kenya.
Real time quality control for phone surveys
When we run phone surveys, we try to ensure that our operators are asking questions and reporting answers in the same way, in order to ensure consistency in the data. However, operator bias happens. For instance, an operator might not be asking questions exactly in the way that they appear in the call script. Or they might misreport what people tell them. Detecting and resolving operator bias helps achieve quality survey results.
Since we started our phone surveys in Iraq earlier this year, we have been able to detect operator bias in near real time. This is because operators enter interview data into a database that we have access to. This means that we are able evaluate operator’s performance as results stream in. We use a regression model that estimates ‘operator effects’ in the survey data. When our model shows that an individual operator’s results vary more than his peers (controlling for many other variables), we suspect that we may have a bias issue, and we try to resolve it by seeking clarifications to the operator.
Many other quality control tools are available for phone surveys. Call centers will record a sample of phone calls. Listening to a few calls helps survey managers understand how the interaction between the agent and the respondent is taking place. Quality assurance is no one-way street: feedback from agents helps flag problems. For instance, operators will tell us when questions are not well understood, and how the call script should to be modified. Deploying a thorough and continuous training program will also support quality. Our training modules and scripts are now online at the mVAM section of the VAM Resource Center.
Kenya: survey calls and helpline, rolled into one
mVAM in Kenya is now in full swing. Since July, our colleagues have been collecting information on food prices by SMS. Also an existing WFP Kenya beneficiary helpline that has been taking incoming calls added outbound calls to its scope of work. The operators are now calling people to gather data on food consumption and coping. Check out our cool new flyer!
That synergies exist between a helpline and an outgoing call survey service is no surprise. From our experience, some simple tips can help maximize these positive effects.
- SMS surveys should include the helpline number in the close-out message. This helps people save the number on their SIM card or phone.
- Survey messages can be used to tell people that calling the helpline is free of charge. This is vital because most people we interact with have no or very little airtime on their phones;
- At the end of mVAM voice calls, operators ask respondents if they have any questions regarding WFP and its programmes.