Alice Clough interviews the VAM team in the Afghanistan country office about why they’ve started using mVAM’s remote food security monitoring system.
In early March, Stephen O’Brien, the United Nations’ Emergency Relief Coordinator, reported that 20 million people across four countries face starvation and famine. The famines looming in Yemen, South Sudan, Somalia and Nigeria represent the largest humanitarian crisis since the UN’s creation. “Without collective and coordinated global efforts,” O’Brien said, “People will simply starve to death, and many more will suffer and die from disease.”
One of the components that complicates these particular emergencies is access to the areas in crisis. Without safe and unimpeded access for humanitarian aid workers, it’s difficult to get a picture of what’s going on in the affected areas, which adds another dimension to an already challenging response. In Northeast Nigeria, the threat of violence made it difficult for WFP’s food security analysts to visit vendors in local markets or speak with people in their homes – all part of their usual food security monitoring routine.
In order to continue gathering information needed to understand the situation in the affected areas, WFP used remote mobile data collection to get a picture of what was happening in the communities they could no longer speak to in person. With an overwhelming amount of responses, we turned to Tableau , who had already helped us create data visualizations for other countries which use mVAM, to help us visualize the results in a way that could be easily understood by everyone.
Our latest interactive data visualization of the food security situation in Northeast Nigeria is now online, and the story of how it came to be can be found on Tableau’s blog. Make sure to check out the free response section, where you can hear from 5,500 households on what should be done to improve the food security in their community.
As some of you might have already guessed, we at mVAM LOVE hackathons. Last weekend, we had a chance to participate in another one: “Hacking Aid” which was organized by the Dutch Ministry of Foreign Affairs and PwC, together with UNHCR, OCHA and Leiden University’s Center of Innovation. This event brought together more than 70 participants from all walks of life – students, aid workers, programmers, developers, linguists, teachers, professionals from the private sector and government. A common thread linked them all: they were brimming with ideas to find digital solutions to some of the pressing challenges the humanitarian community currently faces.
The overall theme of this particular hackathon centered on finding ways to make humanitarian aid more efficient and transparent. Specifically, we looked at solutions that would enable self-reporting by affected populations, so that people in need would be able to report where, when and what type of help is needed.
In order to come up with specific challenges that could be addressed with practical solutions, we had a rapid prototyping session (a.k.a. think hard and quick) to define a problem around collecting and reporting data. This was followed by an open-mic stage, where we pitched our challenges to the hackers.
After working away for nearly 24 hours straight, the twelve teams submitted their final outputs for evaluation by an independent jury panel. The winner of the Hacking Aid award, Team Dream Catchers, developed an app to register complaints and feedback, even offline, from refugees in camps. The second winner, Team Seeing Hunger, proposed a solution to WFP’s challenge: a chatbot tool to pick up and verify self-initiated feedback or reports coming through social media.
A special mention went to Team Botcast, which won the Tech Award for the technically most impressive prototype with their chatbot for Dabanga radio station in Darfur. The chatbot facilitates the process of handling requests for assistance and protection. Team Transformers took home the Innovation Award for their app “Noci”, which uses audiovisual techniques to enable those whom are not able to read or write to report on their needs.
The winners of the hackathon will have a change to travel to Geneva, where they will pitch their ideas to the board of UNHCR, and will receive support from PwC and Leiden University as they develop a prototype. All prototypes will be available open source.
We noticed that many teams proposed chatbot-based solutions to the challenges we pitched, which is exciting for us as it suggests this is a promising area for technological development. mVAM is already exploring how chatbots could be used to help WFP’s work and we hope to find ways to collaborate further with the teams from the hackathon and other partners to vet other ideas in the area of two-way communications.
Late last year, an attack by an armed group on border police posts in Myanmar led to a government security sweep in Rakhine State and recurrent clashes and violence in many villages. As a result, access to a large part of the north of the state was closed off to humanitarian organizations, leaving the already highly vulnerable inhabitants of the townships to fend for themselves.
Unable to access the area since 9 October, WFP decided to use mobile surveys to conduct remote emergency assessments. While not as thorough as face-to-face assessments, mobile surveys could still provide a good snapshot of how people were coping in the areas that were closed off. Furthermore, mobile surveys serve as a means to address a critical information gap where there is little to no information about the needs of the most vulnerable and food insecure, as we have seen in complex emergency settings elsewhere such as during the Ebola crisis and Yemen. But let’s come back to Myanmar and rewind just a few years: hearing from people in these areas would have been impossible – essentially no one had mobile phones.
Myanmar’s mobile transformation
Myanmar’s telecommunication market has come a long way. Not so long ago, Myanmar was one of the “least–connected countries in the world” – just seven years ago, SIM cards cost up to $1,500, and few people had them. In 2013, after the government awarded contracts to two foreign mobile operators, the price of a SIM card fell to $1.50 and network coverage began to roll out across the country. Once the mobile revolution began, things moved fast. Soon, mobile penetration exceeded even that of much better-off neighboring countries, such as Thailand. By 2015, 96 percent of wards and 87 percent of villages in Myanmar had a mobile signal, and nearly 60 percent of households owned a mobile phone.
A case for mobile surveys in Myanmar
WFP’s first mobile assessment in Myanmar took place in November 2016, with 32 key informants from 12 villages in Maungdaw and Buthidaung north, complementing face-to-face interviews of 48 WFP beneficiaries at 8 food distribution points in Buthidaung south. This was at the end of the lean season (the period between harvests when households’ food stocks tend to be the lowest), and respondents told us that due to the deteriorated security situation, people faced serious difficulties in reaching markets, were not able to go to work, nor access agricultural land and fishing areas and. Resulting crop losses could result in mid to long-term impact on food security while households’ terms of trade had decreased and posed a serious concern regarding their ability to purchase sufficient food.
Though low mobile penetration in rural areas of the country posed a challenge for phone surveys, people were nonetheless eager to participate in the survey and share their stories. In order to participate, some people even arranged to borrow phones from neighbors if they did not own one themselves.
A second phone survey in December allowed for a greater sample size and therefore a better understanding of the living conditions in the surveyed areas. WFP spoke to 116 respondents in 70 villages in Maungdaw Township. By this time, the people we spoke with mentioned that there was widespread food insecurity throughout the township. The situation was particularly problematic in the north, where markets were not functioning and access to agricultural land or fishing grounds was restricted. Livelihood opportunities were scarce and the lower demand for daily labour had had an immediate impact on the most vulnerable.
The data collected through the phone surveys helped WFP to get some understanding of the needs in the no-access areas, and to use this information for advocacy with the Government and humanitarian stakeholders. On 9 January 2017, after three months, WFP was finally granted access to all areas where it had operations prior to 9 October, and was able to distribute food to 35,000 people in the villages of Maungdaw north. With the area open again, WFP and its partners are now preparing for thorough assessments on the ground, which will give a fuller picture of the food security situation and also allow us to validate the findings of the phone surveys.
For the second installment of our ‘mind the mode’ series, we’re taking you to Juba, South Sudan, where we previously conducted a mode experiment. What we wanted to see was how food security indicators compare when data is collected face-to-face and through operators over the phone.
South Sudan is a complex setting for mobile surveys to begin with. The country has low cell phone penetration- it’s estimated to be only 20%. Network quality is a problem, often calls don’t go through or audio is poor. Last, but not least, the country has been extremely unstable. While we have been using key informant phone interviews to date, we are investigating the feasibility of conducting phone surveys to collect household food security indicators. Given the complexities, starting with a test to evaluate biases related to survey mode seemed prudent.
The mode experiment took place in “POC 3”, a Protection of Civilians (POC) camp in Juba near the main UN compound. POC 3 is the largest of three camps at the UN House site in Juba, with an estimated population of 20,000 people, according to the International Organization for Migration. People in the POC are there in search of protection against the violence and conflict that South Sudan has been experiencing. We’re hoping to use mobile phones to monitor food security indicators in POC communities. POC 3 happens to have good cell phone coverage – a 2014 survey estimated that some 70% of households in the camp had access to a phone.
We evaluated how mode effects the Food Consumption Score (FCS), which measures the frequency of consumption of different food groups consumed by a household during the 7 days before the survey. A higher score means a better level of the respondent’s household food security. The FCS is a commonly used proxy for household food security.
We carried out two rounds of data collection, round 1 in March and round 2 in May 2016. In round 1, half of the respondents received a voice call survey and the other half participated in an identical interview face-to-face. The ‘treatment’ (voice call) was random. In round 2, some of the respondents that received a voice call took the exact same survey face-to-face, and vice versa.
There were challenges relating to security in the POC and some of the respondents from March were not found in the camp when we conducted the second round in May. As a result, we had 132 voice and 333 face-to-face interviews in round one, but 138 voice and only 117 face-to-face surveys in round 2. This sample size is smaller than we would have liked, but we think it’s indicative enough to tell us how responding to a phone survey differs from one that took place face-to-face.
Calls were placed by operators that were ‘converted’ enumerators – field monitors who usually carry out WFP’s post-distribution monitoring but were new to phone-based surveys. This meant that they were already familiar with the food security indicators and the camp community, but needed training on the protocol for phone-based surveys.
We observed substantial mode effects in round 1. We obtained a mean FCS of 34 via face-to-face surveys, but a much higher score of 45 through voice calls. Our regression analysis shows that mode alone accounted for 7 points in the difference in a household’s response (p<0.01), with other factors accounting for the remainder of the difference. This means that a voice survey would inflate the FCS by 20%, leading to a gross underestimation of the severity of food insecurity in the population of interest. During round 1, the voice FCS question behaved as an almost binary variable – we would get 1s and 7s, but very few 2,3,4,5 answers. That means a lot of people said they ate a given food item one day or every day, but that very few other answers were being recorded.
In round 2, the difference between voice calls and face to face surveys diminished substantially. Also, the difference was not statistically significant. In fact, the slight remaining difference between the two groups was due to respondent households’ socio economic profile, not because of the mode we used to collect data.
For the food consumption score, the differences between voice and face-to-face due to the mode effect were large in round 1, but vanished in round 2. This is a positive finding for us as we are seeking to rigorously test and validate the data collected through mobile and reporting on the results with some degree of confidence. We want to highlight a few lessons here that could help guide others into the right direction.
Lesson 1: Practice makes perfect. We suspect that the poor quality of the data collected in round 1 is due to our call center being brand new, and experiencing ‘teething’ problems. When an in-house call center is first set up, it tends to be small scale comprising of one or two operators. With resources permitting (and provided there is increased information needs) the call center may be expanded with additional operators who will receive regular training and coaching. Our analysts have been saying anecdotally that data quality improves as time goes by and the system becomes more established. We have a good illustration of the phenomenon here in South Sudan.
Lesson 2: Close supervision is required! Although our operators were familiar with data collection, it took time to train them to implement surveys by phone with quality. This again shows that operator selection, training, and supervision are key to obtaining good quality data.
Lesson 3: Work with professional call centers. Overall, this encourages us to continue working with professional call centers when possible, and avoid the temptation to do things in-house in a hurry – something that can be all too tempting in an emergency setting.
We also think the method used in South Sudan could be applied elsewhere to help evaluate mode effects. We will post the survey design on the mVAM Resource Center for others to use.
This week, government, business, academia and civil society leaders will gather at Davos to discuss solutions to the world’s biggest challenges – including how new technologies can be leveraged to solve some of the most serious problems we face. At mVAM, we continue to explore how some of these technologies could be used to help eliminate chronic hunger, malnutrition and food insecurity – most recently looking at how chatbots could help collect important information during a humanitarian response.
Last week, our collaborators at Nielsen – one of the early supporters of mVAM – organized a 24-hour hackathon at the Nielsen Tech Hub in New York City. As part of ongoing efforts through Nielsen Cares, the hackathon aimed to develop an open-source humanitarian chatbot that can collect real-time information about food security. This came at the right time for WFP – we’d developed and tested a prototype of the chatbot with InSTEDD, and Nielsen’s technology and development input helped bring in important new capabilities. Ultimately, our goal is to field-test a chatbot in Haiti in the next few months to help us track food security conditions as people recover from the impacts of Hurricane Matthew.
The event was open to the public. A diverse group of students, volunteer hackers, and Nielsen staff showed up to take on the challenge, despite the wintry weather. InSTEDD’s Director of Platform Engineering, Nicolás di Tada also participated.
Much more than a chatbot
What the hackers built is much more that a chatbot: it is a bona-fide chat-based data collection and reporting system. Rather than attempt to outdo each other (as is the case in most hackathons), the teams split up to build the different components of the system. The different teams, made up of perfect strangers, communicated during the hackathon through Slack. After 24 hours, most components were fully coded up, but there were still bugs with the orchestrator and the gateway that additional post-hackathon work will resolve.
The architecture of the system, as defined by Nielsen, includes:
Leveraging cloud services from the Amazon Web Services product catalog, the teams were able to build a scalable, cost effective platform that can be deployed quickly to multiple locations globally.
Remember the humans
We also received tips from a chatbot specialist, Alec Lazarescu from Chatbots Magazine. He encouraged us to ‘onboard’ users with an initial message that gives people a clear idea of what the chatbot is for. He told us to avoid ‘dead ends’ and allow users to speak to a human being in case they get stuck.
We’re very grateful to Nielsen for their support and to all the participants for their energy and creativity. The next steps involve WFP and InSTEDD accessing the code and work on ironing out the kinks. We expect challenges with the natural language processing in Haitian Creole, a language that is probably under-researched. Making the different parts of the chatbot work together seamlessly also appears to be an area we will still have to work on. And, of course, the final test will be to see whether our target group – people living in Haiti – find the chatbot engaging.
A few months ago, we published a blog post on our plans to test collecting nutrition data through SMS in Malawi and through live voice calls in Kenya. We just got back from Kenya where we conducted a large-scale mode experiment with ICRAF to compare nutrition data collected face-to-face with data collected through phone calls placed by operators at a call center. But before we started our experiment, we did a qualitative formative study to understand rural women’s phone access and use.
We traveled to 16 villages in Baringo and Kitui counties in Kenya, where we conducted focus group discussions and in-depth interviews with women. We also conducted key informant interviews with mobile phone vendors, local nutritionists, and local government leaders.
So in Kenya, can rural women be reached via mobile phone?
Here are the preliminary findings from our qualitative study:
So in conclusion, yes, in Kenya it is possible to reach rural women by phone.
Our findings from Kitui and Baringo counties show that we can reach women in similar contexts with mobile methodologies to collect information on their diet as well as their child’s diet.
We are also analysing the quantitative data from our mode experiment to find out whether data on women and children’s diet collected via live phone operators gives the same results as data collected via traditional face-to-face interviews.
We have success using crowdsourcing to collect food prices by SMS in the refugee camps of Kenya. The experiences made us curious about trying out other methods that could help us deliver data quickly and efficiently from the remote and hard to reach geographies where WFP works.
We found out about a startup that specializes in crowdsourced data collection. Anonymous ‘contributors’ would carry out simple data collection tasks through a dedicated smartphone app, the sales pitch went. Intrigued, we decided to pilot this system to monitor food prices in a drought-affected area of Southern Africa. We were hoping to use the data to complement the information traditional information systems produce. What did we learn?
The anonymous ‘citizen reporter’ is a myth. The company we worked with had to go through local organizations, such as NGOs, to find people able to collect the data for us. This a far cry from the vision of sourcing data from an anonymous crowd. There is more to finding contributors than putting out some ads on social media and magically reaching masses of people. Our contributors were not really anonymous and were easily identified by traders. In the end, the activity looked a lot like traditional tablet-based data collection. The World Bank also found the same thing when they contracted a private company for crowdsourcing. You can find more on their experience in here.
Getting started is labor intensive. It’s going to be a learning process for both your organization and the company, and this will mean investing significant staff time. On our side, since we were unfamiliar with the methodology, there were a lot of iterations as we attempted to specify commodity types and data types. This is perhaps surprising because we at WFP have been collecting food prices for a long time. It turns out we needed to revisit the commodity lists, specify unit measures — a process that required patience. On the company’s side, they had limited experience in the geographies of the pilot which could lead to an overestimation of what was possible and how quickly.
Expect long ramp up times. The ramp up to the data volumes we wanted took months, because that time was needed to set up the system and recruit the local contributors. Our roll out was planned this way. Do not expect an army of anonymous contributors to materialize out of thin air.
It’s still hard to reach remote places. The crowdsourcing model is no silver bullet when it comes to reaching the remote places we were interested in monitoring. It proved hard or even impossible to source enough data from the more remote markets when using a crowdsourcing service. This is perhaps because of low smartphone penetration in remote locations, the high cost of sending a contributor to such places, or to poor connectivity. In contrast, collecting data from larger urban areas was much easier.
High costs are a barrier to handover in resource-poor environments. It became clear that the cost of the activity was higher than lower-tech alternatives. WFP works to enable handover of information systems to national authorities or other local partners. For the moment, the cost of app-based crowdsourcing is perhaps out of the financial reach of our local partners.
After some trial and error, we were able to obtain good quality data through crowdsourcing that was helpful to our field offices. However, ultimately we returned to our mVAM strategy – using phone calls to traders to collect food prices each week. Although our approach can’t cover as many commodities as the company’s crowdsourcing activity provided, it has its own advantages. It’s lower tech – there is no fancy app to download. There is no far away company to deal with. Above all, it’s an approach we can hand over to our local partners.
1. Awards for Remote Mobile Data Collection Work
At the Humanitarian Technology 2016 conference, our paper Knowing Just in Time ‘Knowing Just in Time’ won Best Paper for Outstanding Impact. In the paper, we assessed mVAM’s contribution to decision-making by looking at use cases for mVAM in camps, conflict settings and vulnerable geographies. Check out our blog Tech for Humanity for more on it and our other conference paper mVAM: a New Contribution to the Information Ecology of Humanitarian Work
To close the year, we had a nice surprise from Nominet Trust, the UK’s leading tech for good funder. We made their 100 most inspiring social innovations using digital technology to drive social change around the world.
2. New Tech
We’re also in the era of Big Data. We partnered with Flowminder, experts in analyzing call detail records, to track displacement in Haiti after Hurricane Matthew. Find out more in ‘After the storm: using big data to track displacement in Haiti’
We’re also super excited about the chatbot we started developing for messaging apps and our roll out of Free Basics in Malawi which is allowing us to share the food prices we collect in mVAM surveys with people in Malawi With mVAM, our main focus has been reaching people on their simple feature phones. But we know that smartphone ownership is only going to increase. Contacting people through internet-enabled phones opens up loads of new forms of communication and data collection. is still reaching people on their -free basics
mVAM expanded to 16 new countries facing a wide set of challenges: conflict, El Nino drought, hurricanes, extremely remote geographies. We’ve been tracking and learning about what remote mobile data collection can add to food security monitoring systems and what its limits are in different contexts. For some of the highlights, check out our blogs on Afghanistan, Democratic Republic of Congo, Haiti, Nigeria, Papua New Guinea, and El Nino in Southern Africa,
4. Dynamic Partnerships
To have a lasting impact, we need to work with governments. We are really proud of our partnership with CAID, the Cellule d’Analyses des Indicateurs du Développement under the Prime Minister’s Office in the Democratic Republic of Congo. We collaborated on setting up a national market monitoring system- mKengela that they are now running. We’ve had intensive technical sessions with the CAID team in Rome and Kinshasa to work on solutions that will fit their data management and analysis needs. The CAID team even traveled to Johannesburg to share their remote mobile data experience with other African countries and help other governments use this technology.
We’re also working with Leiden University. Bouncing ideas off of their team at the Centre for Innovation helps us move forward on tricky challenges. We’re also collaborating with them to develop an online course where we’re going to share our methodologies and how to use remote technology to monitor food security. Check out Welcome to Vamistan for more.
We are in the field of tech. So we can’t do our job well without partnering with the private sector. It’s definitely a dynamic area, and also one where we at mVAM are learning what works best in melding our humanitarian goals with the exciting private tech potential out there. Check out our blog From the Rift Valley to Silicon Valley and our hackathon with Data Mission for more.
5. Learning- the neverending process
In addition to trying out new technology, we’ve been trying to answer some important questions about the live calls, SMS, and IVR surveys which make up the bulk of mVAM data collection. We’re also doing mode experiments to understand how people answer differently based on which mode we use to contact them. Check out our first Mind the Mode article with more coming in 2017. In Kenya, we are looking into whether we can ask nutrition indicators through mVAM methods. A major challenge is reaching women through phone surveys so we organized a gender webinar with partners to learn from what they are doing- check out our key gender takeaways. These are key questions and they can’t be resolved overnight. But we’re making steady progress in understanding them, and we’re excited for what more we’ll find out in 2017.
Thanks to everyone who has supported our work this year and kept up with our blog!