A new mVAM baby in Mali, weight: 7800 respondents!

WFP/Sebastien Rieussec

WFP/Sebastien Rieussec

This week we’re reporting on our latest news from mVAM in Mali. In this landlocked country in the Sahel chronic food insecurity and malnutrition is widespread – WFP has been present in Mali since 1964. In the last few years Mali has been coping with numerous shocks – such as droughts, floods and a military coup – that led to a political and security crisis and increased food insecurity in the country: by 2016 around 3.1 million people in Mali were food insecure. Households are particularly affected during the lean season, between June and September; and this year WFP estimated 3.8 million people affected by food insecurity, of which 601,00 people in urgent need of food assistance.

To monitor the food security situation, the Government of Mali, with WFP support, does two nationwide face-to-face surveys, in February and September each year. However, in between these times and especially during the lean season that takes place during the summer in Mali there was no data collection – so mVAM was there to fill the ‘data gap.’ We’ve previously blogged about the Mali mode experiment we did comparing data collected by live calls and face-to-face data. As the results showed that there was little difference between the modes, in August the Country Office rolled out mVAM nationwide so that they could get food security information from households affected by this particularly difficult period of the year. During the previous face-to-face survey phone numbers were collected…out of the 13,400 numbers we collected we reached over 7,800 households – mVAM’s largest-ever survey!

With each survey comes different country-specific ‘problems’. There are many different reasons why people might not want to take part in a phone survey – but in Mali, we found one of the biggest was mistrust. People are not used to doing surveys via mobile phones and are sure that there is some form of trick behind them. Many reported that they know that there are lots of mobile phone scams and worry that the call from an unknown number purporting to be from WFP is just another one of these. One of the reasons why they were suspicious  was due to the fact that there was a long time gap between the number collection and the phone survey. This was actually a deliberate choice by the Country Office to ensure that the survey was not just a ‘follow up’ survey to face-to-face data collection like our mode experiment and was getting new information during this specific time period. What wasn’t foreseen was that this meant people forgot that they had given WFP their number and may have not fully understood why they did so in the first place.

Mali blog Edith 2

WFP/Nanthilde Kamara

To get around this issue, the Country Office is planning to use several tactics. As well as using SMS and national radio to advertise the survey, the next time that phone numbers are collected, there will be more time spent on explaining exactly what the purpose of the survey is. The annual September face-to-face food security survey is currently ongoing, so enumerators are now explaining that they might be called by WFP later on this year. The call centre that supports mVAM in Mali calls everyone with the same unique number, this number will be shared with community leaders just before the survey so that they can inform people that they will be rung by this specific number and that it’s an official call from WFP. Respondents will then be able to save the number in their phone so they know when they get the call exactly who it is and it won’t be just an unknown number.

The analysis is still ongoing: We’re looking forward to the results!

 

Mind the mode …. and the non-response

How voice and face-to-face survey data compares in Mali

This is the third entry in our ‘Mind the Mode’ series on the mVAM blog. We are constantly assessing our data collection modalities to better understand what produces the most-accurate results and what biases may be present. One of our recent experiments took us to Mali, where we were comparing the food consumption score between face-to-face (F2F) interviews versus mVAM live calls.

It’s all in the details
To do this, in February and March, the WFP team first conducted a baseline assessment in four regions of the country. As part of the baseline, we collected phone numbers from participants. Approximately 7-10 days later, we then re-contacted those households who had phones, reaching roughly half of those encountered during the face-to-face survey. We weren’t able to contact the other households. To ensure the validity of the results, we made sure the questionnaire was the exact same between the F2F and telephone interviews. Any differences in wording or changes in the way in which the questions were asked could adversely affect our analysis.

The findings from our analysis were quite interesting. We found that food consumption scores (FCS) collected via the mVAM survey tended to be slightly higher than those collected via the face-to-face survey. The graph below illustrates this shift to higher scores between the two rounds. Higher FCS via mVAM versus F2F surveys is not atypical to Mali. We’ve observed similar outcomes in South Sudan and other countries where mVAM studies have taken place.

mali dist

 

Why could this be? There are two main reasons that could explain this difference. Either it might be due to the data collection modality (i.e., people report higher food consumption scores on the phone)? Or, a perhaps a selection bias is occurring? Remember that we were only able to contact roughly half of the participants from the F2F survey during the telephone calls. So, it’s possible that people who responded to the phone calls are less food insecure, which could make sense, since we often see that the poorest of the poor either don’t own a phone or have limited economic means to charge their phone or purchase phone credit.

To test these hypotheses, we dug a bit deeper.

Same same…
Are people telling the same story on the phone versus face-to-face? Based on our results, the answer is yes! If we compare the same pool of respondents who participated in both the F2F and telephone survey rounds, their food security indicators are more or less the same. For example, the mean mVAM FCS was 56.21 while the mean F2F FCS was 55.65, with no statistically significant difference between the two.

But different…
So what about selection bias? In the F2F round, there are essentially three groups of people: 1) those who own phones and participated in both the F2F and mVAM survey; 2) people who own phones but didn’t participate in the mVAM survey, because they either didn’t answer the calls or their phone was off; and 3) people who do not own a phone and thus couldn’t participate in the mVAM survey.

People who replied to the mVAM survey have overall higher FCS than those that we were unable to contact. What we learned from this experiment is that bias does not only come from the households that do not own a phone but also from non-respondents (those households who shared their phone number and gave consent but then were not reachable later on for the phone interview). Possible reasons why they were not reachable could be that they have less access to electricity to charge their phone or that they live in areas with bad network coverage. The graph below illustrates the distribution by respondent type and their respective FCS.

mali boxp

When you compare the demographics of people in these three groups based on the data collected in the baseline, you can see that there are significant differences, as per the example below. Notice that the education levels of respondents varies amongst the three groups—those without a phone tend to be less educated than those who own a phone and participated in the mVAM survey.

mali profile

This study taught us a valuable lesson. While we are confident that there is no statistically significant difference between face-to-face and phone responses within the Mali context, there is a selection bias in mVAM-collected data. By not including those without phones as well as those who did not respond, we are missing an important (and likely poorer) subset of the population, meaning that the reported FCS is likely higher than it may be if these groups were included. One way to account for this bias is to ensure that telephone operators attempt to contact the households numerous times, over the course of several days. It’s important that they really try to reach them. The team is also studying how to account for this bias in our data analyses.

Postcard from Bangui

Good to be OKING:It may not be new and super large, but the owner claims this phone has a week-long battery life! WFP/Dominique Ferretti

It may not be new and super large, but the owner claims this phone has a week-long battery life!
WFP/Dominique Ferretti

Greetings from the Central African Republic (CAR)! Our team recently visited Bangui and Kaga-Bandoro to help the Country Office team assess how to enhance the current mVAM system and see what other mVAM technologies we might be able to deploy. CAR is a very unique context, because there’s little-to-no cell phone reception outside of main towns. Only 26% of the population own a phone, one of the lowest rates in the world according to the World Bank.  This means that collecting data remotely takes some creativity. The CAR team uses a key informant system, where they contact approximately 200 people around the country each month to collect information on basic commodity prices, market access, population movements, and security issues. The collected information is then shared with the humanitarian community, who appreciate the data, as it’s the only national-level food security data that’s currently collected regularly!

A local woman in Kaga-Bandoro selling a great source of protein and a central African delicacy—caterpillars! WFP/Dominique Ferretti

A local woman in Kaga-Bandoro selling a great source of protein and a central African delicacy—caterpillars!
WFP/Dominique Ferretti

The only downfall to the key informant system is that it doesn’t give us household-level food security information. The CAR team has therefore decided to try a small pilot using household questionnaires in the city of Kaga-Bandoro. Courtesy of UNHAS, we visited the city (more like a very small town!) and the 2 IDP camps it hosts during our day trip. While not that many people had cell phones, enough community members and displaced persons had phones that we’ll be able to get some idea of the food security situation.

Stay tuned for more as the pilot unfolds…!

Chatting with community members as they collect water WFP/Dominique Ferretti

Chatting with community members as they collect water
WFP/Dominique Ferretti

If you’re not human then who are you?

Experimenting with chatbots in Nigeria and Haiti

WFP/Lucia Casarin

Testing the bot in Haiti – WFP/Lucia Casarin

Readers of this blog know that the team has been experimenting with chatbots to communicate with disaster-affected communities – read our previous posts about our prototype and the Nielsen Hackathon.

As part of this effort, during recent missions to Haiti and Nigeria, our team went out to talk to communities to find out whether a chatbot would be right for them.

Would a chatbot be a stretch in these communities?

Well it’s not that much of a stretch.

In North East Nigeria, most displaced people live in Maiduguri, a city of over 1 million people. In this ‘urban’ setting connectivity is good, most people own cell phones and many young people use social media and messaging apps. Mobile operators have been offering services that allow people to access the internet by selling ‘social bundles’ (unlimited social media access sold in very small increments) and offer some services for free, including Facebook Light and Facebook Messenger.

In Haiti, three-quarters of the population live in the capital, Port-au-Prince, where 3G connectivity is good and most people use messaging apps to communicate with friends and family. Even in rural and difficult-to-reach communities, leaders and young people own smartphones and connect to the internet. There is a lot of competition between mobile operators so the prices for mobile data are very low. This means that most people can afford to access the internet either via their own smartphone or from shared smartphones.

A

Mobile phones charging station on the road from Léogane Peri to Port-au-Prince WFP/Lucia Casarin

A bare-bones demo

In both countries we tested a simple chatbot that asks people about food prices and what the food security is like in their community. The survey we used was much more basic than our usual mobile questionnaires as we felt it was important to keep things simple at this stage.

For Nigeria, the bot demo was initially in English but we soon translated it into Hausa, the primary language spoken by displaced persons in Maiduguri. In Haiti we made it available both in Creole and French. The chatbot was very responsive on 3G and it even worked with slower 2G connections so the technology works in these contexts. But this was only the starting point, what we really wanted to know was what ‘real’ people thought about the bot.

We organized focus group discussions with displaced people in Maiduguri and with community representatives in Haiti. We helped people access the WFP bot via their Facebook accounts, and they began chatting away.

Sounds cool, but what are the limitations?

Here’s what people said:

First of all, people thought the bot is a convenient, quick, and easy way to get in touch directly with WFP and they really liked that the bot allows them to speak to WFP without intermediaries. They had lot to tell us particularly through the open-ended question where they typed out detailed responses.

In Nigeria, they did tell us that our (somewhat wordy) English-language demo should be translated into Hausa because it would make it easier for everyone to use. Our first group of testers were young people who were already Facebook users and so were familiar with Messenger. It was therefore no surprise that they were interacting smoothly with the bot and able to go through our questionnaire in minutes.

WFP/Jean-Martin Bauer

Testing the bot in Nigeria – WFP/Jean-Martin Bauer

In Haiti, people started interacting with the bot as if it was a human rather than an automated questionnaire so they got stuck pretty fast when it wasn’t as naturally responsive as they’d expected. This means that either we give clearer instructions to people or we add Natural Language Processing capabilities to our bot.

There are of course other barriers. In both countries women appeared to be less likely to own a smartphone. This means that bot users will likely be overwhelmingly young, male and better educated than other people – hardly ‘representative’ of WFP’s target affected population. The free version of the bot is also not always available: in Nigeria only Airtel subscribers can access it, while in Haiti the free service doesn’t exist yet.

This means that the bot would need to be a complement to the other tools we have. We might use data from the bot to obtain a quick situation update, but we will continue relying on other sources for more representative data.

Hearing from those who are #FacingFamine

Photo: WFP/Amadou Baraze

Photo: WFP/Amadou Baraze

 

In early March, Stephen O’Brien, the United Nations’ Emergency Relief Coordinator, reported that 20 million people across four countries face starvation and famine.  The famines looming in Yemen, South Sudan, Somalia and Nigeria represent the largest humanitarian crisis since the UN’s creation. “Without collective and coordinated global efforts,” O’Brien said, “People will simply starve to death, and many more will suffer and die from disease.”

One of the components that complicates these particular emergencies is access to the areas in crisis. Without safe and unimpeded access for humanitarian aid workers, it’s difficult to get a picture of what’s going on in the affected areas, which adds another dimension to an already challenging response. In Northeast Nigeria, the threat of violence made it difficult for WFP’s food security analysts to visit vendors in local markets or speak with people in their homes – all part of their usual food security monitoring routine.

In order to continue gathering information needed to understand the situation in the affected areas, WFP used remote mobile data collection to get a picture of what was happening in the communities they could no longer speak to in person. With an overwhelming amount of responses, we turned to Tableau , who had already helped us create data visualizations for other countries which use mVAM, to help us visualize the results in a way that could be easily understood by everyone.

mVAM hears directly from people in affected communities in the northeast of Nigeria

mVAM hears directly from people in affected communities in the northeast of Nigeria

 

Our latest interactive data visualization of the food security situation in Northeast Nigeria is now online, and the story of how it came to be can be found on Tableau’s blog. Make sure to check out the free response section, where you can hear from 5,500 households on what should be done to improve the food security in their community.