As mVAM has been expanding we’ve started to see the remote technology used in other areas of WFP’s work. This week’s blog is from Katy Huang who works for the Yemen Country Office. She shares how the M&E unit is using remote live calls to get feedback from beneficiaries about WFP’s assistance.
Before joining WFP Yemen’s Monitoring and Evaluation (M&E) unit 8 month ago, I worked as a researcher for the New York City Health Department. As I love the creative process of collecting, analyzing and reporting data, I was excited for the opportunity to manage our unit’s “remote M&E” (rM&E) system. Currently, Yemen’s Emergency Operation assists about 3 million beneficiaries a month. Our rM&E system uses an third country call center to conduct phone surveys with beneficiaries post-distribution to hear about their experiences receiving and using the assistance. The center completes about 2000-2400 surveys per month.
Before establishing the rM&E system in September 2015, we learned in a previous post-distribution monitoring survey that a large majority of our emergency in-kind food beneficiaries owned a mobile phone or had access to a friend or neighbor’s mobile. We also found out that a large majority of mobile owners were able to charge their mobiles on a regular basis. This information meant that conducting mobile surveys proved to be ideal within the context of Yemen’s ongoing and widespread conflict as it allowed us to reach large numbers of beneficiaries without compromising the safety of field monitors. Other benefits of using rM&E include it’s relative low cost and being we can reach beneficiaries in all the governorates where we offer assistance. Also, in the 15-20 minutes it takes to complete a survey, we have been able to collect all the key process and food security outcome indicators that we also collect in our longer bi-annual face-to-face post-distribution monitoring surveys. Ultimately, rM&E complements other M&E systems (i.e., on-site distribution monitoring and beneficiary hotline) to triangulate and confirm findings.
Although there are many benefits to using rM&E, challenges do exist. Bias is the main issue as data collected by rM&E tends to be more biased than data collected face-to-face. Some of the biases we face relate to the following:
- Sampling frame bias: We don’t have the entire list of mobile numbers of beneficiaries for random calling. The amount of mobile numbers we receive depends on what cooperating partners collect from beneficiaries at the time of food distribution. We have had to regularly remind cooperating partners about the importance of sending us these mobile lists. In addition, some beneficiaries don’t own a mobile phone and they may have different characteristics, such as being more poor or vulnerable, than those that do own mobiles.
- Gender bias: The frequency of female respondents for rM&E (about 5 percent) are lower than that of face-to-face (about 10 percent). This may be due to more males than females owning mobile phones. To try to address this, the call center recently hired more female enumerators to engage female beneficiaries to respond.
Despite these biases, the amount and quality of data we have been able to collect on a monthly basis have been invaluable. The large sample size has allowed us to report nationally representative data and to disaggregate data by activity type (i.e., in-kind, voucher) or demographics (i.e., displacement, gender). With regular monitoring, we are able to see trends and compare results over months and quarters. To see how we used this data for reporting, please see our Yemen M&E Quarter 1 2016 report.
For more information on mVAM’s work in Yemen, please visit the mVAM Yemen site.