Episode 11: 20 June 2017
Jean-Martin Bauer and Seokjin Han travel to North-Eastern Nigeria to test out our humanitarian chatbot with IDPs affected by the Boko Haram insurgency,
Greetings from an ever-green Juba! The last time we reported from South Sudan it was dry and dusty everywhere. This time our visit coincided with the start of the rainy season – a welcome respite to the scorching heat that lasted for months.
Other than the heat, there are many challenges in South Sudan, particularly when trying to set up an mVAM system. South Sudan is one of the worst ranking countries in terms of mobile phone penetration and connectivity: according to 2016 ITU data, approximately 24 percent of the population have mobile cellular subscriptions and merely 4.3 percent of households own a computer. The ongoing conflict has only made the situation worse. We found out that network coverage has significantly deteriorated since mVAM activities first started in February 2016. A case in point: one major network operator, which reportedly had the largest outreach in the country, reduced its coverage from 70 to 15 percent. Our mVAM operators told us that completing a 10-minute survey with one single phone call was nearly impossible, because the line is constantly dropping.
Even when a call does go through, it is extremely difficult to pinpoint the respondent’s location. People are on the move fleeing the conflict (more than 950,000 South Sudanese have crossed the border into Uganda alone according to the latest UNHCR estimates) and phone numbers keep changing (the average shelf life of a SIM is short as people are on the move and network coverage varies greatly between different areas). To make things even more complicated, the administrative boundaries of the country are also shifting (in addition to the existing 10 states, an additional 22 states have been newly created).
Being mindful of these challenges, we had previously recommended that the country office start contacting a pool of key informants who are easier to reach and were able to collect data on markets, displacement, and road access in the Greater Upper Nile Region. However, even here we are confronted with the challenge of collecting data in a highly politically-divided context. Relying exclusively on key informant sources can give you a biased picture of the situation on the ground, especially where the informants speak for specific interest groups. It is therefore necessary to triangulate various sources of key informant information and complement them with other secondary or even primary household data when possible.
Does all of this mean that there is no future for mVAM in South Sudan? On the contrary, we found that the demand for mobile surveys is there both for WFP and the humanitarian community at large. After all, South Sudan is a complex emergency where ‘putting boots on the ground’ is often not possible and we need all the creativity and tools we can muster. In fact, WFP South Sudan has been conducting mobile surveys for market monitoring and rapid emergency food security assessments (the latest one took place in select famine-affected counties). Similarly, other NGO and development partners on the ground are also conducting mobile surveys for programme or food security monitoring.
Moving forward, we have identified, together with the South Sudan VAM team, two areas of opportunity where we can scale mVAM: i) urban food security monitoring in selected hotspots and interest points and ii) complementing the early warning bulletin jointly produced by the Ministry of Humanitarian Affairs and Disaster Management and WFP with mVAM key informant data.
This is the most recent entry in our ‘Mind the Mode’ series on the mVAM blog. We are constantly assessing our data collection modalities to better understand what produces the most-accurate results and what biases may be present. One of our recent experiments took us to Mali, where we were comparing the food consumption score between face-to-face (F2F) interviews versus mVAM live calls.
It’s all in the details
To do this, in February and March, the WFP team first conducted a baseline assessment in four regions of the country. As part of the baseline, we collected phone numbers from participants. Approximately 7-10 days later, we then re-contacted those households who had phones, reaching roughly half of those encountered during the face-to-face survey. We weren’t able to contact the other households. To ensure the validity of the results, we made sure the questionnaire was the exact same between the F2F and telephone interviews. Any differences in wording or changes in the way in which the questions were asked could adversely affect our analysis.
The findings from our analysis were quite interesting. We found that food consumption scores (FCS) collected via the mVAM survey tended to be slightly higher than those collected via the face-to-face survey. The graph below illustrates this shift to higher scores between the two rounds. Higher FCS via mVAM versus F2F surveys is not atypical to Mali. We’ve observed similar outcomes in South Sudan and other countries where mVAM studies have taken place.
Why could this be? There are two main reasons that could explain this difference. Either it might be due to the data collection modality (i.e., people report higher food consumption scores on the phone)? Or, a perhaps a selection bias is occurring? Remember that we were only able to contact roughly half of the participants from the F2F survey during the telephone calls. So, it’s possible that people who responded to the phone calls are less food insecure, which could make sense, since we often see that the poorest of the poor either don’t own a phone or have limited economic means to charge their phone or purchase phone credit.
To test these hypotheses, we dug a bit deeper.
Are people telling the same story on the phone versus face-to-face? Based on our results, the answer is yes! If we compare the same pool of respondents who participated in both the F2F and telephone survey rounds, their food security indicators are more or less the same. For example, the mean mVAM FCS was 56.21 while the mean F2F FCS was 55.65, with no statistically significant difference between the two.
So what about selection bias? In the F2F round, there are essentially three groups of people: 1) those who own phones and participated in both the F2F and mVAM survey; 2) people who own phones but didn’t participate in the mVAM survey, because they either didn’t answer the calls or their phone was off; and 3) people who do not own a phone and thus couldn’t participate in the mVAM survey.
People who replied to the mVAM survey have overall higher FCS than those that we were unable to contact. What we learned from this experiment is that bias does not only come from the households that do not own a phone but also from non-respondents (those households who shared their phone number and gave consent but then were not reachable later on for the phone interview). Possible reasons why they were not reachable could be that they have less access to electricity to charge their phone or that they live in areas with bad network coverage. The graph below illustrates the distribution by respondent type and their respective FCS.
When you compare the demographics of people in these three groups based on the data collected in the baseline, you can see that there are significant differences, as per the example below. Notice that the education levels of respondents varies amongst the three groups—those without a phone tend to be less educated than those who own a phone and participated in the mVAM survey.
This study taught us a valuable lesson. While we are confident that there is no statistically significant difference between face-to-face and phone responses within the Mali context, there is a selection bias in mVAM-collected data. By not including those without phones as well as those who did not respond, we are missing an important (and likely poorer) subset of the population, meaning that the reported FCS is likely higher than it may be if these groups were included. One way to account for this bias is to ensure that telephone operators attempt to contact the households numerous times, over the course of several days. It’s important that they really try to reach them. The team is also studying how to account for this bias in our data analyses.
Over the last ten months we have been testing if we can use mobile phones to collect nutrition indicators. One of these experiments involved using SMS to ask questions about women’s diet quality via the Minimum Dietary Diversity – Women (MDD-W) indicator. The MDD-W involves asking simple questions about whether women of reproductive age (15-49 years) consumed at least five out of ten defined food groups. We were interested in using SMS surveys to measure MDD-W, because SMS offers an opportunity to collect data regularly at scale and at low cost.
From October 2016 to April 2017, we worked with GeoPoll to conduct five survey rounds on MDD-W and find a way to adapt the indicator to SMS. We analysed data from each round, identified gaps and refined the survey instrument. We were able to collect data quickly and identify strengths and weaknesses to make revisions through an iterative process. Through this process, we believe that we have successfully designed an instrument that can be used to monitor MDD-W trends by SMS. Here’s a short summary of what we learned:
1. Using a mix of open-ended and list-based questions helped people better understand our questions.
By using a mix of open-ended and list-based questions, we were able to significantly improve data quality. In the first few rounds, we had an unusually high number of respondents who either scored “0” or “10” on the MDD-W score, which are both unlikely under normal circumstances. A score of “0” means that the respondent did not consume food items from any of the 10 core food groups the previous day or night, while a score of “10” means that the respondent consumed food items from all food groups. In the first round, scores of “0” or “10” accounted for 29 percent of all respondents, but by Round 5 these scores represented only 3 percent of responses. It seems that having respondents reflect about what they ate in the open-ended questions we introduced in later rounds helps them recall the food items they consumed and answer the subsequent list-based questions more accurately.
2. Keep questions simple.
We originally asked people by SMS whether they ate food items from the core food groups that comprise the MDD-W score. For example, “Yesterday, did you eat any Vitamin A-rich fruits and vegetables such as mangos, carrots, pumpkin, …….” Perhaps respondents thought that they needed to consume food items from both the fruit and vegetable groups in order to reply “yes” to this question. So instead, we split that question into two separate questions (one on Vitamin A-rich fruits and the other on Vitamin A-rich vegetables) to make it easier for the respondent to answer. We did the same for some of the other questions and found a very low percentage of women scoring “0” or “10” on the MDD-W score. Of course there is a trade-off here, and splitting too many questions might lead to a long and unwieldy questionnaire that could frustrate respondents.
3. Let respondents take the survey in their preferred language.
Comprehension remains a challenge in automated surveys, so helping respondents by asking questions in their own language will ensure data quality and limit non-response. In the Malawi study, translating food items into the local language (Chichewa), while keeping the rest of the questionnaire in English, improved comprehension. We recommend providing the respondent with the option to take the survey in their preferred language.
4. Pre-stratify and pre-target to ensure representativeness.
SMS surveys tend to be biased towards people who have mobile phones; we reach a lot of younger, urban men, and relatively few women of reproductive age, our target group for surveys on women’s diet. To ensure we are reaching them, an MDD-W SMS survey should be designed or ‘pre-stratified’ to include a diverse group of respondents. In Malawi, we were able to pre-stratify according to variables that included age, level of education, location and wealth. This allowed us to include women from all walks of life.
5. Post-calibrate to produce estimates that are more comparable to face-to-face surveys.
The MDD-W SMS surveys we conducted produced higher point estimates than those we would expect in face-to-face surveys. This suggests we may wish to consider calibration to adjust for sampling bias, the likely cause for the discrepancy. Calibration is the process of maintaining instrument accuracy by minimizing factors that cause inaccurate measurements. We’re still working on this and hope to find a solution soon. In the meantime, we think we are able to track trends in MDD-W by SMS with some reliability.
Greetings from the Central African Republic (CAR)! Our team recently visited Bangui and Kaga-Bandoro to help the Country Office team assess how to enhance the current mVAM system and see what other mVAM technologies we might be able to deploy. CAR is a very unique context, because there’s little-to-no cell phone reception outside of main towns. Only 26% of the population own a phone, one of the lowest rates in the world according to the World Bank. This means that collecting data remotely takes some creativity. The CAR team uses a key informant system, where they contact approximately 200 people around the country each month to collect information on basic commodity prices, market access, population movements, and security issues. The collected information is then shared with the humanitarian community, who appreciate the data, as it’s the only national-level food security data that’s currently collected regularly!
The only downfall to the key informant system is that it doesn’t give us household-level food security information. The CAR team has therefore decided to try a small pilot using household questionnaires in the city of Kaga-Bandoro. Courtesy of UNHAS, we visited the city (more like a very small town!) and the 2 IDP camps it hosts during our day trip. While not that many people had cell phones, enough community members and displaced persons had phones that we’ll be able to get some idea of the food security situation.
Stay tuned for more as the pilot unfolds…!
The use of mobile technology is a tremendous opportunity to better communicate with people in humanitarian settings. However, these advanced capabilities also involve new privacy and security risks for people in the communities where remote mobile surveys are implemented. We therefore collaborated with the International Data Responsibility Group and Leiden University’s Centre for Innovation to draft a practical guide: ‘Conducting Mobile Surveys Responsibly: A field book for WFP staff’.
The field book outlines the main risks for staff engaged in remote data collection and details best practices for data security, privacy and responsible data approaches in the very complex environments in which WFP operates.
As part of this effort, during recent missions to Haiti and Nigeria, our team went out to talk to communities to find out whether a chatbot would be right for them.
Would a chatbot be a stretch in these communities?
Well it’s not that much of a stretch.
In North East Nigeria, most displaced people live in Maiduguri, a city of over 1 million people. In this ‘urban’ setting connectivity is good, most people own cell phones and many young people use social media and messaging apps. Mobile operators have been offering services that allow people to access the internet by selling ‘social bundles’ (unlimited social media access sold in very small increments) and offer some services for free, including Facebook Light and Facebook Messenger.
In Haiti, three-quarters of the population live in the capital, Port-au-Prince, where 3G connectivity is good and most people use messaging apps to communicate with friends and family. Even in rural and difficult-to-reach communities, leaders and young people own smartphones and connect to the internet. There is a lot of competition between mobile operators so the prices for mobile data are very low. This means that most people can afford to access the internet either via their own smartphone or from shared smartphones.
A bare-bones demo
In both countries we tested a simple chatbot that asks people about food prices and what the food security is like in their community. The survey we used was much more basic than our usual mobile questionnaires as we felt it was important to keep things simple at this stage.
For Nigeria, the bot demo was initially in English but we soon translated it into Hausa, the primary language spoken by displaced persons in Maiduguri. In Haiti we made it available both in Creole and French. The chatbot was very responsive on 3G and it even worked with slower 2G connections so the technology works in these contexts. But this was only the starting point, what we really wanted to know was what ‘real’ people thought about the bot.
We organized focus group discussions with displaced people in Maiduguri and with community representatives in Haiti. We helped people access the WFP bot via their Facebook accounts, and they began chatting away.
Sounds cool, but what are the limitations?
Here’s what people said:
First of all, people thought the bot is a convenient, quick, and easy way to get in touch directly with WFP and they really liked that the bot allows them to speak to WFP without intermediaries. They had lot to tell us particularly through the open-ended question where they typed out detailed responses.
In Nigeria, they did tell us that our (somewhat wordy) English-language demo should be translated into Hausa because it would make it easier for everyone to use. Our first group of testers were young people who were already Facebook users and so were familiar with Messenger. It was therefore no surprise that they were interacting smoothly with the bot and able to go through our questionnaire in minutes.
In Haiti, people started interacting with the bot as if it was a human rather than an automated questionnaire so they got stuck pretty fast when it wasn’t as naturally responsive as they’d expected. This means that either we give clearer instructions to people or we add Natural Language Processing capabilities to our bot.
There are of course other barriers. In both countries women appeared to be less likely to own a smartphone. This means that bot users will likely be overwhelmingly young, male and better educated than other people – hardly ‘representative’ of WFP’s target affected population. The free version of the bot is also not always available: in Nigeria only Airtel subscribers can access it, while in Haiti the free service doesn’t exist yet.
This means that the bot would need to be a complement to the other tools we have. We might use data from the bot to obtain a quick situation update, but we will continue relying on other sources for more representative data.
We’ve used mVAM to collect data on a range of things that impact food security – so what about information on nutrition? Back in October, we went to Kenya to conduct a study on whether we could use remote mobile data collection to gather information on women and children’s nutrition.
We wrote to you back in November about one of our new innovations – our Free Basics website ‘Za Pamsika’ where we’re posting commodity prices using the weekly price data we’re collecting through our mVAM operators on a free website. We said that the project had the potential to reach millions of Malawians – well, a lot has happened since then.
Rather than continuing to willfully upload prices while watching our user statistics go up and down, we went to Malawi to carry out a short ground truth study and get some first hand user feedback. The aim of the mission was to investigate the best way of using the website and interrogate the assumptions we’d made when designing it.
With this in mind, we tried to answer two big questions:
So we went to rural and urban markets in the Central and Southern regions to speak to the mVAM traders and the consumers in their markets about their mobile phone usage and market activity and to get their feedback on the website.
What kind of answers did we get?
First – access issues. While you don’t need a smartphone to access the website we knew that mobile penetration in Malawi is low. So we were most worried about the prevalence of internet-enabled phones and network coverage. From our study we found out that while we aren’t going to be able to reach everyone in Malawi via a website, we can still communicate with people. Network coverage was a problem in some areas. However, overall we found that most of the traders had internet enabled phones or wanted to buy one. We also found that Malawi’s MNOs have been recently trying to push out better network coverage. All good news for future reach of the website.
Actually the biggest barrier was language and literacy. While English is the national language of Malawi, most of the literate people we spoke to were much more comfortable reading and writing in Chichewa because that’s what they were taught in. While they were very enthusiastic about the website content when it was explained to them, they found the initial design (all in English and text heavy) confusing and difficult to use. Luckily this is an easy change to make so we did a quick redesign of the website and translated it into Chichewa:
With our new design we headed back into the markets and got much better feedback. Rather than just saying that they liked the website content they could really interact with it and were making comments on the different maize and beans prices.
The second barrier we found was digital literacy. Many of the people we spoke to had internet-enabled phones but either didn’t know how to use them or didn’t even realise that they had the internet on them! Unlike the language change this is not a quick fix. This was particularly prominent amongst the women we spoke to, none of whom were comfortable with mobile internet. We’re therefore going to partner with civil society organisations promoting digital literacy. WFP has a network of partners and farmers on the ground who they reach out to with climate information so we’re going to try and use these focal points to communicate our prices with vulnerable populations and communities who have limited access to information.
But do Malawians really want a ‘Za Pamsika’ website?
It turned out that maize and beans prices really are something that people want to see on the website. The recent drought was on everyone’s minds and they were really emphasising how much of a difference getting a good price could make. People were also already using their phones to get prices – by calling their friends or other traders in different areas and were quite enthusiastic about the possibility of getting this information for free.
With these learnings in mind and feeling confident with our website redesign and excited to be working closely with the country office, we embarked on our next steps. We now have a new focal point in the Lilongwe office who’s looking after the project and in a much better placed position than us in Rome to reach out to millions of Malawians. By this point over 25,000 people had already visited the Za Pamsika website but we knew our reach could be much further. We therefore started experimenting with ways of advertising the website.
First – we decided to take out a Facebook ad to try and raise the site profile so we created our own ‘Za Pamsika’ page on Facebook and put out some ads in English and Chichewa. We were pretty excited when they started showing up on Malawian colleagues’ Facebook newsfeeds and within 10 days we’d reached more than 130,000 people and got 650 likes to our Facebook page.
What we didn’t expect was the organic reaction we’d get to our page. Within 3 days we’d not only reached more than 80,000 people with our post, we’d also seen that people started having conversations about maize prices on our advert. People have also started messaging us about whether we can add their market to our website. We’re also getting comments about what other commodities we should add, for instance more seasonal foods such as groundnuts or soya. Most excitingly we even had someone knock on the door of the sub-office to inquire about the website after seeing our advert!
On a second mission in April we went out to the markets in Lilongwe again armed with our new ‘Za Pamsika’ posters. We were putting them up in the trader’s shops and were pretty quickly swamped with people excited about the website and how it could save them money. But again – everyone was asking us to add more food prices to the site – it seems like Malawians just keep wanting to know more about ‘things you find in the market’!
So what’s next for Za Pamsika?
We’ve got our new focal point Khataza on board who’s taking charge of the website. First up, taking requests into account, we will be adding other seasonal commodities to the website. We’re going to continue experimenting with our Facebook ads and start using our Facebook page to reach out and engage with people about what they’d like on the page. We’ve also got some new partnerships coming up with civil society organisations who are keen to spread the word about ‘Za Pamsika’ and who we can work with to break down access barriers to this information.
Are millions in Malawi being reached? Not yet – but we’re getting there.
One of the major challenges we currently face is that while our survey results provide a detailed picture of the food security situation at the regional level, they are only able to provide representative food security estimates at a larger geographic scale, and don’t always tell us where smaller hotspots or pockets of food insecurity are. So we want to find a way to produce the most accurate, up-to-date and granular representations of food insecurity as possible, to help inform our decision making.
Recently some of our team had the great chance to go to Southampton – a peaceful city in the south of the UK – where we loaded up on shortbread and started working on a type of dynamic high-resolution mapping known as Geostatistical Mapping.
The purpose of the trip was to work with and learn from Flowminder/WorldPop. As you might remember, we’ve worked with them in the past to do things like tracking population displacement in Haiti after Hurricane Matthew. They’ve also developed a way to produce high-resolution maps of population demographics and characteristics. We believe these methods can be applied to create high resolution maps of food security indicators.
As modelling techniques and data processing capability have evolved, and as high resolution satellite imagery has become more available, creating more granular maps than ever before is possible. This is where Flowminder/WorldPop comes into play. Their aim is to provide estimates of population demographics and characteristics for low and middle income countries by integrating census, survey, satellite and GIS datasets, in a flexible machine-learning framework.
So, how does it work? (if you’re not a satistician, skip to the pictures!)
Basically, these high-resolution maps use one or more geolocated data sets, such as rainfall, vegetation or accessibility to markets, and look at the correlation between these secondary sources of geospatial data and something else, say, a particular food security indicator from a household survey in sampled areas (for this reason, high resolution mapping is also referred to as geospatial mapping) . Once we understand the relationship between the two variables in sampled areas, we can make more accurate predictions about the food security situation in non-sampled areas. If available, mobile phone metadata (Call Detail Records) can also be used as an additional covariate, especially in urban areas where the mobile network is dense.
Looking at the example above and the difference in coverage, you’ve probably already understood how appealing high-resolution maps are as a tool for better planning. But we don’t want to stop here – we’re young and full of dreams! If you noticed, we spoke at the beginning of this post about dynamic high resolution maps. We just discussed how to get a static map for more detailed spatial information, but the next step is actually to update this map each time we have new data. This is a great opportunity, because some satellite imagery already provides new data every ten days or so. This means that we could have maps representing the situation in near real-time.
To take this step, we have to bring in data that is available on a high-frequency basis, such as mobile surveys. These can be used to highlight some areas of our map on regular basis, or to assess the accuracy of the map by checking hotspots with a quick mobile survey.