How are respondents selected?
mVAM identifies respondents in three ways:
- by asking respondents of traditional face-to-face surveys to agree to a follow-up phone survey
- by randomly calling people through mobile phone user rolls who have volunteered to take phone surveys. Telecom companies maintain a list of phone numbers of subscribers who agree to participate in surveys. Randomly selected mobile phone users in the areas of interest to WFP are then contacted, as per our sampling instructions.
- by calling numbers generated through random digit dialing
Respondents are always given the choice to opt in to the survey or decline.
Are there any costs involved for respondents to participate in the mobile surveys?
There are no costs for respondents. Incoming calls are free, so taking a live call or an IVR survey costs nothing to the respondent. For SMS, answers to our survey questions are routed through a short code, which means all telecommunication costs are billed to WFP. The short code works like a toll-free number.
Do we provide airtime credit incentives to survey participants in all countries?
Airtime credit incentives are provided to mVAM respondents upon successful completion of a survey in almost all of our survey operations. The only exception is for mVAM surveys for Syria and Yemen, where mVAM is working with an out of country call center based in Amman, Jordan and contacting households by generating telephone numbers using random digit dialing, and therefore unable to transfer airtime credit incentives to participants. However, more than material incentives, we find that altruism is the biggest driver of response
How do we ensure that the data is reliable?
Representative sampling and using stratification, drawing inferences from large enough samples, and thorough identification of key informants.
Also, in order to assess data quality, whenever possible WFP implements a concurrent face-to-face survey to robustly evaluate the quality of the data that remote data collection surveys produce. When conditions do not allow a face-to-face baseline (for instance: high insecurity, restrictions on movement), WFP evaluates data quality with reference to existing sources of baseline data.
Is the data representative?
mVAM data tends to be geographically representative. Before data collection a target sample size will be set which will determine the level of reporting results for the results. After data has been collected the analysts will look at the confidence intervals and run statistical tests to detect statistically significant changes between data collection rounds.
When data from a recent census or a comprehensive household survey (such as the DHS) is available, mVAM collects one or several socio-demographic proxy indicators from the interviewed households that match with an existing question in the comprehensive dataset. For example in Burundi, this question is the roof type of the respondent household and a post-stratification process is then applied to re-weight the data.
Is there a bias and how does mVAM account for it?
Due to uneven mobile phone ownership and penetration rates at different levels (geographic/ household) some bias in mobile phone survey results is inherent; results may be biased towards respondents that are relatively wealthier, younger, more literate, male and from urban areas with acceptable mobile coverage.
Thus, WFP attempts to profile respondents by wealth and focus capturing and monitoring food security trends over time, and early warning functions, rather than estimating food insecurity levels. Also. Also, results are weighed by population and number of phones owned, and triangulated with other sources of information to avoid errors of interpretation. In order to understand who is replying to our surveys, we obtain information on each respondent’s demographics and socio- economic status, either through a prior face-to-face baseline survey or by asking ‘profiling’ questions. We account for bias when we analyse data: for instance, if our respondents are skewed towards women (as they have been in DR Congo and Somalia), results can be reweighed to reflect the composition of the population using a correction factor. It is worth noting that with the spectacular growth of mobile coverage (17 percent per annum, according to the GSMA), mobile phone surveys are rapidly becoming more robust.
Language can also introduce bias, as well as operators who conduct the surveys. In the case of multiple local languages, data collection is carried out in the most commonly spoken local languages to the extent that is feasible Also, if surveys were conducted through call centre operators, unintentional operator bias will be assessed.
Additional methodology documents are available on the country pages on the mVAM site.