Episode 11: 20 June 2017
Jean-Martin Bauer and Seokjin Han travel to North-Eastern Nigeria to test out our humanitarian chatbot with IDPs affected by the Boko Haram insurgency,
Greetings from an ever-green Juba! The last time we reported from South Sudan it was dry and dusty everywhere. This time our visit coincided with the start of the rainy season – a welcome respite to the scorching heat that lasted for months.
Other than the heat, there are many challenges in South Sudan, particularly when trying to set up an mVAM system. South Sudan is one of the worst ranking countries in terms of mobile phone penetration and connectivity: according to 2016 ITU data, approximately 24 percent of the population have mobile cellular subscriptions and merely 4.3 percent of households own a computer. The ongoing conflict has only made the situation worse. We found out that network coverage has significantly deteriorated since mVAM activities first started in February 2016. A case in point: one major network operator, which reportedly had the largest outreach in the country, reduced its coverage from 70 to 15 percent. Our mVAM operators told us that completing a 10-minute survey with one single phone call was nearly impossible, because the line is constantly dropping.
Even when a call does go through, it is extremely difficult to pinpoint the respondent’s location. People are on the move fleeing the conflict (more than 950,000 South Sudanese have crossed the border into Uganda alone according to the latest UNHCR estimates) and phone numbers keep changing (the average shelf life of a SIM is short as people are on the move and network coverage varies greatly between different areas). To make things even more complicated, the administrative boundaries of the country are also shifting (in addition to the existing 10 states, an additional 22 states have been newly created).
Being mindful of these challenges, we had previously recommended that the country office start contacting a pool of key informants who are easier to reach and were able to collect data on markets, displacement, and road access in the Greater Upper Nile Region. However, even here we are confronted with the challenge of collecting data in a highly politically-divided context. Relying exclusively on key informant sources can give you a biased picture of the situation on the ground, especially where the informants speak for specific interest groups. It is therefore necessary to triangulate various sources of key informant information and complement them with other secondary or even primary household data when possible.
Does all of this mean that there is no future for mVAM in South Sudan? On the contrary, we found that the demand for mobile surveys is there both for WFP and the humanitarian community at large. After all, South Sudan is a complex emergency where ‘putting boots on the ground’ is often not possible and we need all the creativity and tools we can muster. In fact, WFP South Sudan has been conducting mobile surveys for market monitoring and rapid emergency food security assessments (the latest one took place in select famine-affected counties). Similarly, other NGO and development partners on the ground are also conducting mobile surveys for programme or food security monitoring.
Moving forward, we have identified, together with the South Sudan VAM team, two areas of opportunity where we can scale mVAM: i) urban food security monitoring in selected hotspots and interest points and ii) complementing the early warning bulletin jointly produced by the Ministry of Humanitarian Affairs and Disaster Management and WFP with mVAM key informant data.
This is the most recent entry in our ‘Mind the Mode’ series on the mVAM blog. We are constantly assessing our data collection modalities to better understand what produces the most-accurate results and what biases may be present. One of our recent experiments took us to Mali, where we were comparing the food consumption score between face-to-face (F2F) interviews versus mVAM live calls.
It’s all in the details
To do this, in February and March, the WFP team first conducted a baseline assessment in four regions of the country. As part of the baseline, we collected phone numbers from participants. Approximately 7-10 days later, we then re-contacted those households who had phones, reaching roughly half of those encountered during the face-to-face survey. We weren’t able to contact the other households. To ensure the validity of the results, we made sure the questionnaire was the exact same between the F2F and telephone interviews. Any differences in wording or changes in the way in which the questions were asked could adversely affect our analysis.
The findings from our analysis were quite interesting. We found that food consumption scores (FCS) collected via the mVAM survey tended to be slightly higher than those collected via the face-to-face survey. The graph below illustrates this shift to higher scores between the two rounds. Higher FCS via mVAM versus F2F surveys is not atypical to Mali. We’ve observed similar outcomes in South Sudan and other countries where mVAM studies have taken place.
Why could this be? There are two main reasons that could explain this difference. Either it might be due to the data collection modality (i.e., people report higher food consumption scores on the phone)? Or, a perhaps a selection bias is occurring? Remember that we were only able to contact roughly half of the participants from the F2F survey during the telephone calls. So, it’s possible that people who responded to the phone calls are less food insecure, which could make sense, since we often see that the poorest of the poor either don’t own a phone or have limited economic means to charge their phone or purchase phone credit.
To test these hypotheses, we dug a bit deeper.
Are people telling the same story on the phone versus face-to-face? Based on our results, the answer is yes! If we compare the same pool of respondents who participated in both the F2F and telephone survey rounds, their food security indicators are more or less the same. For example, the mean mVAM FCS was 56.21 while the mean F2F FCS was 55.65, with no statistically significant difference between the two.
So what about selection bias? In the F2F round, there are essentially three groups of people: 1) those who own phones and participated in both the F2F and mVAM survey; 2) people who own phones but didn’t participate in the mVAM survey, because they either didn’t answer the calls or their phone was off; and 3) people who do not own a phone and thus couldn’t participate in the mVAM survey.
People who replied to the mVAM survey have overall higher FCS than those that we were unable to contact. What we learned from this experiment is that bias does not only come from the households that do not own a phone but also from non-respondents (those households who shared their phone number and gave consent but then were not reachable later on for the phone interview). Possible reasons why they were not reachable could be that they have less access to electricity to charge their phone or that they live in areas with bad network coverage. The graph below illustrates the distribution by respondent type and their respective FCS.
When you compare the demographics of people in these three groups based on the data collected in the baseline, you can see that there are significant differences, as per the example below. Notice that the education levels of respondents varies amongst the three groups—those without a phone tend to be less educated than those who own a phone and participated in the mVAM survey.
This study taught us a valuable lesson. While we are confident that there is no statistically significant difference between face-to-face and phone responses within the Mali context, there is a selection bias in mVAM-collected data. By not including those without phones as well as those who did not respond, we are missing an important (and likely poorer) subset of the population, meaning that the reported FCS is likely higher than it may be if these groups were included. One way to account for this bias is to ensure that telephone operators attempt to contact the households numerous times, over the course of several days. It’s important that they really try to reach them. The team is also studying how to account for this bias in our data analyses.
The use of mobile technology is a tremendous opportunity to better communicate with people in humanitarian settings. However, these advanced capabilities also involve new privacy and security risks for people in the communities where remote mobile surveys are implemented. We therefore collaborated with the International Data Responsibility Group and Leiden University’s Centre for Innovation to draft a practical guide: ‘Conducting Mobile Surveys Responsibly: A field book for WFP staff’.
The field book outlines the main risks for staff engaged in remote data collection and details best practices for data security, privacy and responsible data approaches in the very complex environments in which WFP operates.
Have you ever wanted to help out the World Food Programme? Sign up for our Chatbot Hackathon!
When: Our partner Nielsen is holding a 24 hour ‘Hack for Hunger’ Chatbot Hackathon from Saturday, January 7th to on Sunday, January 8, 2017 at their global headquarters. The Hackathon is sponsored by Nielsen, the world’s largest data and information company, and their data science subsidiary eXelate and Nielsen Marketing Cloud. You can sign up at this link
The Challenge: Build an ‘emergency response chatbot’ to revolutionize how we get the information we need to respond during emergencies.
The United Nations World Food Programme is the world’s largest humanitarian agency fighting hunger. When there is a crisis, we get the bags of food or cash assistance to people to make sure they don’t go hungry. We currently assist around 80 million people in 80 countries around the world. But to do this well, we need to know where people are that need the most help.
The chatbot you build will allow community members to report to WFP about food security conditions in their local area. This information can save lives after a disaster like Hurricane Matthew in Haiti where roads were destroyed and ports were closed for days. WFP can chat with community members and find out what is happening on the ground in order to get assistance to the areas that need it most.
Our Chief Economist Arif Husain will be at the hackathon to tell you more about the World Food Programme’s work in emergencies and technology’s potential to accelerate our response. For more info on work we’ve been doing on chatbots with InStedd, read our blog: chatbot prototype
So, are you up for the challenge?
We have said it before: open data is not really useful unless it’s also accessible to everyone. WFP maintains an extensive food price database that is accessed by a lot of people, but most of our visitors happen to be donors or agencies in North America and Europe. We feel that in order to achieve Zero Hunger, information needs to be accessible for people living in the most vulnerable geographies.
We’ve already experimented with ways to share food price information with vulnerable communities using SMS and IVR in Somalia and DRC. Recently, an interesting opportunity came up: sharing our data through Facebook’s new internet platform Free Basics. This initiative aims to provide the 2/3 of the world’s population who do not have internet with basic web access and is currently available in 53 countries. Certain websites with “basic” content like news, employment opportunities, health, education and local information are available for free with no data charges. We’re always interested in exploring how new technology could help in the fight against hunger, so this month, we began testing our first Free Basics website in Malawi.
Malawi: Our first test for Free Basics.
Why did we choose Malawi? Along with other countries in southern Africa, Malawi was affected by a drought that has affected agricultural production and caused food prices to soar. In late 2015, WFP set up a phone based market monitoring system that helps us track food prices all over the country on a weekly basis. Current forecasts estimate that 6.5 million people won’t be able to meet their basic food requirements. Households at risk of food insecurity can spend anywhere from half to three quarters of their budget on food, so sharing the data we have about food prices with the population might help people make informed decisions about their food purchases.
Introducing ‘Za Pamsika’
We’ve been working with the Praekelt Foundation incubator to set up a Free Basics website that shares this weekly price data for all of the districts we get the data from. We’ve called it ‘Za Pamsika’ literally ‘things you can find in the market’ in Chichewa, the main language in Malawi. It’s a really simple website. You click on your region and district to find out food prices in your area. You can also compare prices at nearby markets if you’re in an area with many market options.
The great thing about Free Basics is that you don’t need a smartphone to access our data for free – just an internet-enabled one from a participating MNO. The project therefore has the potential to provide food price information to all Malawians who have mobile phones with internet browsing capabilities. It’s also not even necessary to have a Facebook account! While we know that by only contacting people who have internet-enabled phones, we may be missing the most vulnerable households. But it will still provide useful information for a large section of the population. Essentially, we see it as a step in the right direction toward making our data accessible to everyone.
The site has now been live for 10 days, and we’ve started seeing some results coming in, both the number of visitors and the demographics. As we’d expected, most of the people who’ve visited so far have been male and under 25.
It’s great to see that we already have some users, but we still have to make sure more people are aware of the site and it’s useful for them. We’ll be heading to the markets covered in the site with the WFP Country Office team, to speak to people first hand about the site and learn how we can improve it. Ultimately, the feedback we receive from people on the ground will help us to evaluate Free Basics as a tool to share data about food with the communities we serve.
From time to time, people interested in mVAM will suggest we distribute cell phones to people who live in the vulnerable and food insecure communities where we work. It is usually a well-meaning idea, originating from specialists as they scope out a mobile-based data collection activity or a donor trying to help. After all, while phone
ownership has expanded exponentially, many poor people still do not have phones. It’s tempting to distribute devices to them as a solution to this disparity. However, our experience at mVAM and elsewhere has shown that we’re often better off going with the phones that people already have. This blog post explains why.
A checklist from UNICEF innovation
Others working with technology in developing countries have faced the same debates. We found that UNICEF innovation has come up with a useful list of questions to think about before distributing handsets or other hardware.
An addendum from mVAM
On the basis of our experience, we’d like to add a few additional considerations to this list. mVAM’s own experience with cell phone distribution has been mixed. In 2012, we provided 400 cell phones to IDPs in eastern DR Congo, an idea we picked up from the literature (e.g. Listening to Dar). We tried to do the right thing by consulting with the community first. We offered low-end feature phones. We set up a solar charging station in the camp where people could recharge their phones for free. We also ensured that people received training in using the new devices.
On the one hand, we were pleased to see that we obtained a good response rate to our surveys from the camp and that providing access to cell phone technology empowered people (because the cell phones we provided allowed displaced people to call home, use mobile money for remittances, and obtain information). On the other hand, there unfortunately was also theft (42 phones were reported stolen!) and even cases of people being attacked for their devices. These findings are captured in the independent review of the mVAM activity that was published in 2015. Due to these concerns, we have not provided cell phones to people in other settings as mVAM has expanded.
So, we want to add another set of considerations to the UNICEF list :
As we have gained more experience working in different contexts, we have concluded that we’re better off working with the phones people already have. Sometimes, that means designing our data collection strategy around the information that can be credibly collected when phone ownership or network coverage is limited. For instance, in places where cell phone penetration is low (~20%), we have not attempted to run a representative survey of households but have focused instead on obtaining information from a set of key informants, as we did in Afghanistan or the Central African Republic.
As mVAM has been expanding we’ve started to see the remote technology used in other areas of WFP’s work. This week’s blog is from Katy Huang who works for the Yemen Country Office. She shares how the M&E unit is using remote live calls to get feedback from beneficiaries about WFP’s assistance.
Before joining WFP Yemen’s Monitoring and Evaluation (M&E) unit 8 month ago, I worked as a researcher for the New York City Health Department. As I love the creative process of collecting, analyzing and reporting data, I was excited for the opportunity to manage our unit’s “remote M&E” (rM&E) system. Currently, Yemen’s Emergency Operation assists about 3 million beneficiaries a month. Our rM&E system uses an third country call center to conduct phone surveys with beneficiaries post-distribution to hear about their experiences receiving and using the assistance. The center completes about 2000-2400 surveys per month.
Before establishing the rM&E system in September 2015, we learned in a previous post-distribution monitoring survey that a large majority of our emergency in-kind food beneficiaries owned a mobile phone or had access to a friend or neighbor’s mobile. We also found out that a large majority of mobile owners were able to charge their mobiles on a regular basis. This information meant that conducting mobile surveys proved to be ideal within the context of Yemen’s ongoing and widespread conflict as it allowed us to reach large numbers of beneficiaries without compromising the safety of field monitors. Other benefits of using rM&E include it’s relative low cost and being we can reach beneficiaries in all the governorates where we offer assistance. Also, in the 15-20 minutes it takes to complete a survey, we have been able to collect all the key process and food security outcome indicators that we also collect in our longer bi-annual face-to-face post-distribution monitoring surveys. Ultimately, rM&E complements other M&E systems (i.e., on-site distribution monitoring and beneficiary hotline) to triangulate and confirm findings.
Although there are many benefits to using rM&E, challenges do exist. Bias is the main issue as data collected by rM&E tends to be more biased than data collected face-to-face. Some of the biases we face relate to the following:
Despite these biases, the amount and quality of data we have been able to collect on a monthly basis have been invaluable. The large sample size has allowed us to report nationally representative data and to disaggregate data by activity type (i.e., in-kind, voucher) or demographics (i.e., displacement, gender). With regular monitoring, we are able to see trends and compare results over months and quarters. To see how we used this data for reporting, please see our Yemen M&E Quarter 1 2016 report.
For more information on mVAM’s work in Yemen, please visit the mVAM Yemen site.