Postcard from Dakar

mVAM workshop participants all smiles after learning more about IVR WFP/Lucia Casarn

mVAM workshop participants all smiles after learning more about IVR WFP/Lucia Casarn

During the last week of June, staff from WFP HQ’s mVAM team, the West and Central Africa Regional Bureau, and Nigeria and Niger Country Offices met in beautiful Dakar to work together on Interactive Voice Response (IVR) systems for two-way communication. (If you want to dig deep into all details IVR-related, check out the lesson in our mVAM online course!)

We’ve previously blogged about how WFP is responding to the needs of people who have been displaced due to Boko Haram insurgencies in both Nigeria and Niger. When we implemented these operations we also put communication channels in place so beneficiaries are able to contact WFP. In Nigeria, the Maiduguri Field Office created a hotline. Their operators receive an average of 100 calls per day from beneficiaries asking food security-related questions and providing feedback on the operations. The problem is the hotline is only available during working hours and has a limited number of people who can call in at the same time. To work around this they’re therefore looking at how an IVR system can support the call operators who are dealing with high volumes and better manage calls that take place outside of normal office hours. WFP Niger wants to set up a similar hotline system but without full time phone operators. Beneficiaries will call in to an automated IVR system and their queries and feedback recorded by the system and followed up by the Country Office. 

A Nigeria IT Officer working to install a GSM gateway for IVR usage in Maiduguri WFP/Lucia Casarin

A Nigeria IT Officer working to install a GSM gateway for IVR usage in Maiduguri WFP/Lucia Casarin

During the workshop participants were trained by InSTEDD on how to physically deploy IVR using a GSM gateway (a fancy tool that automatically places phone calls) and Verboice, the free open source software they’ve developed to manage these systems. The team also discussed the nitty gritty technical aspects of the system, including creating and modifying call flows (the sequencing of questions), scheduling calls and downloading collected call logs and recordings. Most importantly, participants had the opportunity to share their experiences and challenges with experts in this field and discuss best practices, alternative deployments and technical solutions.

The Country Office staff have now returned to Niger and Nigeria and they’ve already started testing the use of the IVR machines. We’re eager to begin logging data and hearing more from our beneficiaries. So stay tuned!

 

 

 

Postcard from DRC

Congo Call Center operator on the phone with a WFP beneficiary to discuss WFP’s activities

Congo Call Center operator on the phone with a WFP beneficiary to discuss WFP’s activities

Greetings from the Democratic Republic of Congo (DRC)! Two members of the mVAM team recently travelled to Kinshasa to help the WFP Country Office assess how to improve upon its current mVAM system and see what other mVAM technologies we could roll out in the coming months.

mVAM data collection in DRC is conducted nation-wide in collaboration with the Cellule d’Analyses des Indicateurs de Développement, more commonly referred to as CAID. CAID is part of the Congolese National Government, housed within the Prime Minister’s Office, and is responsible for collecting food security and other indicators on a regular basis. Since April 2016, CAID—with technical support from WFP—has been collecting remote food security data across more than 50% of the country. Now this is quite a feat when you consider the vast size of DRC (it would cover most of Western Europe!) coupled with the fact that in many places there is little network coverage.

Map from: http://www.mylifeelsewhere.com/country-size-comparison/belgium/democratic-republic-of-congo

Map from: http://www.mylifeelsewhere.com/country-size-comparison/belgium/democratic-republic-of-congo

During the visit, the mVAM team met with CAID to discuss how to improve its data quality and expand to areas not yet covered by mVAM. This included a visit to the Congo Call Center (CCC). They have a team of operators dedicated to conducting the monthly calls so we went to discuss any challenges they encounter when placing calls. We also brainstormed different ways share the information that CAID collects with the general public. They currently produce a monthly bulletin called ‘m-kengela’ that shares price information and other food security-related details but they also want to share this information with a larger audience. So, together we explored the possibility of creating a Free Basics website that would be accessible to a larger audience. Given the success of our Free Basics pilot in Malawi and the fact that there are two participating mobile network providers within the country, we decided that this would be an ideal way of creating a nation-wide price website. We therefore met with cell phone companies and spent time with CAID mapping out what their site might look like.

Discussions are now underway vis-a-vis the next steps and the mVAM team and CAID are hard at work preparing for the launch of its first Free Basics price website. So stay tuned for more details as rollout takes place over the coming months!

mVAM falls in love in Nepal

WFP/Gaurav Singhal

The mVAM-NeKSAP team (WFP/Gaurav Singhal)

 

The mVAM Team is on the move again. This time our travels took us to Kathmandu, Nepal, where we’re not only excited at the prospect of using mVAM for the first time, but mVAM has also fallen in love and is soon be a proud parent!

The Government of Nepal currently runs a key-informant based food security monitoring system it calls NeKSAP. Each trimester, community leaders in 74 of 75 of Nepal’s Districts gather and use convergence evidence to assess the criticality of the food-security situation in their respective area. This exercise has proven invaluable in directing programming and resources not just for WFP but also for the government and other development/humanitarian organizations across the country. But the process is cumbersome and a bit imprecise when it comes to understanding and responding to a vastly complicated humanitarian landscape (think about the 2015 earthquake in Kathmandu and climate change!).

In 2016, an extreme drought in the high-altitude plateaus of the Mid-Western and Far-Western Development Regions of the country prompted the WFP Nepal Country Office to conduct a face-to-face food security baseline assessment. Given the persistent acute food insecurity in the region the team, collaborating with the Government of Nepal, requested assistance to create a seasonal food security monitoring system, leveraging the agility, efficiency, and cost-effectiveness of mVAM.

This region is so remote that in the past WFP preferred to use airlifts than trucks to deliver assistance. This could not have been a bigger challenge for mVAM given issues with selection bias and phone-ownership. This was compounded by the fact that government insisted on nothing less than high-quality, representative, publishable statistics that could be used in official government figures. If only there was some way we had to reach the most remote and inaccessible regions as well that can only be reached by travelling for several days on foot! As it turns out, the NeKSAP system has provisions for a small network of skilled enumerators to live and work in these regions.

Rural landscape in Bajhang District, located in the North Western region of the country WFP/Bikkil Sthapit

Rural landscape in Bajhang District, located in the North Western region of the country
WFP/Bikkil Sthapit

That is when we had brainwave borrowed from South Asian tradition: an arranged marriage for mVAM (don’t worry, there was a courtship first)…and mNeKSAP was conceived combining the best of traditional face-to-face assessment with mVAM! Why only rely on one survey mode?  For individuals without phones we decided to use the NeKSAP enumerators to do traditional face-to-face assessments.

Furthermore, all the individuals were first interviewed in a face-to-face pre-winter baseline. This means that not only is the data representative, ensuring coverage of non-phone owners, mNeKSAP also provides a rare panel data set, re-interviewing the same individuals every trimester over a year.  Panel data is the gold-standard for doing causal inference. There is much more work to be done of course but we’ll keep you updated on this exciting new collaboration.

VAM Talks Episode 11: IDPs in Nigeria test our chatbot

Episode 11:   20 June 2017

Logo2Jean-Martin Bauer and Seokjin Han travel to North-Eastern Nigeria to test out our humanitarian chatbot with IDPs affected by the Boko Haram insurgency,

After the rains: sprouts of green and mVAM in South Sudan

Our South Sudan mVAM operators at work (WFP/Angie Lee)

Our South Sudan mVAM operators at work (WFP/Angie Lee)

Greetings from an ever-green Juba! The last time we reported from South Sudan it was dry and dusty everywhere. This time our visit coincided with the start of the rainy season – a welcome respite to the scorching heat that lasted for months.

Other than the heat, there are many challenges in South Sudan, particularly when trying to set up an mVAM system. South Sudan is one of the worst ranking countries in terms of mobile phone penetration and connectivity: according to 2016 ITU data, approximately 24 percent of the population have mobile cellular subscriptions and merely 4.3 percent of households own a computer. The ongoing conflict has only made the situation worse. We found out that network coverage has significantly deteriorated since mVAM activities first started in February 2016. A case in point: one major network operator, which reportedly had the largest outreach in the country, reduced its coverage from 70 to 15 percent. Our mVAM operators told us that completing a 10-minute survey with one single phone call was nearly impossible, because the line is constantly dropping.

Even when a call does go through, it is extremely difficult to pinpoint the respondent’s location. People are on the move fleeing the conflict (more than 950,000 South Sudanese have crossed the border into Uganda alone according to the latest UNHCR estimates) and phone numbers keep changing (the average shelf life of a SIM is short as people are on the move and network coverage varies greatly between different areas). To make things even more complicated, the administrative boundaries of the country are also shifting (in addition to the existing 10 states, an additional 22 states have been newly created).

Being mindful of these challenges, we had previously recommended that the country office start contacting a pool of key informants who are easier to reach and were able to collect data on markets, displacement, and road access in the Greater Upper Nile Region. However, even here we are confronted with the challenge of collecting data in a highly politically-divided context. Relying exclusively on key informant sources can give you a biased picture of the situation on the ground, especially where the informants speak for specific interest groups. It is therefore necessary to triangulate various sources of key informant information and complement them with other secondary or even primary household data when possible.

Does all of this mean that there is no future for mVAM in South Sudan? On the contrary, we found that the demand for mobile surveys is there both for WFP and the humanitarian community at large. After all, South Sudan is a complex emergency where ‘putting boots on the ground’ is often not possible and we need all the creativity and tools we can muster. In fact, WFP South Sudan has been conducting mobile surveys for market monitoring and rapid emergency food security assessments (the latest one took place in select famine-affected counties). Similarly, other NGO and development partners on the ground are also conducting mobile surveys for programme or food security monitoring.

Moving forward, we have identified, together with the South Sudan VAM team, two areas of opportunity where we can scale mVAM: i) urban food security monitoring in selected hotspots and interest points and ii) complementing the early warning bulletin jointly produced by the Ministry of Humanitarian Affairs and Disaster Management and WFP with mVAM key informant data.

Green fellows like these are often found at WFP premises (WFP/Marie Enlund)

Green fellows like these are often found at WFP premises (WFP/Marie Enlund)

Mind the mode …. and the non-response

How voice and face-to-face survey data compares in Mali

This is the third entry in our ‘Mind the Mode’ series on the mVAM blog. We are constantly assessing our data collection modalities to better understand what produces the most-accurate results and what biases may be present. One of our recent experiments took us to Mali, where we were comparing the food consumption score between face-to-face (F2F) interviews versus mVAM live calls.

It’s all in the details
To do this, in February and March, the WFP team first conducted a baseline assessment in four regions of the country. As part of the baseline, we collected phone numbers from participants. Approximately 7-10 days later, we then re-contacted those households who had phones, reaching roughly half of those encountered during the face-to-face survey. We weren’t able to contact the other households. To ensure the validity of the results, we made sure the questionnaire was the exact same between the F2F and telephone interviews. Any differences in wording or changes in the way in which the questions were asked could adversely affect our analysis.

The findings from our analysis were quite interesting. We found that food consumption scores (FCS) collected via the mVAM survey tended to be slightly higher than those collected via the face-to-face survey. The graph below illustrates this shift to higher scores between the two rounds. Higher FCS via mVAM versus F2F surveys is not atypical to Mali. We’ve observed similar outcomes in South Sudan and other countries where mVAM studies have taken place.

mali dist

 

Why could this be? There are two main reasons that could explain this difference. Either it might be due to the data collection modality (i.e., people report higher food consumption scores on the phone)? Or, a perhaps a selection bias is occurring? Remember that we were only able to contact roughly half of the participants from the F2F survey during the telephone calls. So, it’s possible that people who responded to the phone calls are less food insecure, which could make sense, since we often see that the poorest of the poor either don’t own a phone or have limited economic means to charge their phone or purchase phone credit.

To test these hypotheses, we dug a bit deeper.

Same same…
Are people telling the same story on the phone versus face-to-face? Based on our results, the answer is yes! If we compare the same pool of respondents who participated in both the F2F and telephone survey rounds, their food security indicators are more or less the same. For example, the mean mVAM FCS was 56.21 while the mean F2F FCS was 55.65, with no statistically significant difference between the two.

But different…
So what about selection bias? In the F2F round, there are essentially three groups of people: 1) those who own phones and participated in both the F2F and mVAM survey; 2) people who own phones but didn’t participate in the mVAM survey, because they either didn’t answer the calls or their phone was off; and 3) people who do not own a phone and thus couldn’t participate in the mVAM survey.

People who replied to the mVAM survey have overall higher FCS than those that we were unable to contact. What we learned from this experiment is that bias does not only come from the households that do not own a phone but also from non-respondents (those households who shared their phone number and gave consent but then were not reachable later on for the phone interview). Possible reasons why they were not reachable could be that they have less access to electricity to charge their phone or that they live in areas with bad network coverage. The graph below illustrates the distribution by respondent type and their respective FCS.

mali boxp

When you compare the demographics of people in these three groups based on the data collected in the baseline, you can see that there are significant differences, as per the example below. Notice that the education levels of respondents varies amongst the three groups—those without a phone tend to be less educated than those who own a phone and participated in the mVAM survey.

mali profile

This study taught us a valuable lesson. While we are confident that there is no statistically significant difference between face-to-face and phone responses within the Mali context, there is a selection bias in mVAM-collected data. By not including those without phones as well as those who did not respond, we are missing an important (and likely poorer) subset of the population, meaning that the reported FCS is likely higher than it may be if these groups were included. One way to account for this bias is to ensure that telephone operators attempt to contact the households numerous times, over the course of several days. It’s important that they really try to reach them. The team is also studying how to account for this bias in our data analyses.

Trial and Error: How we found a way to monitor nutrition through SMS in Malawi

WFP/Alice Clough

WFP/Alice Clough

Over the last ten months we have been testing if we can use mobile phones to collect nutrition indicators. One of these experiments involved using SMS to ask questions about women’s diet quality via the Minimum Dietary Diversity – Women (MDD-W) indicator.  The MDD-W involves asking simple questions about whether women of reproductive age (15-49 years) consumed at least five out of ten defined food groups. We were interested in using SMS surveys to measure MDD-W, because SMS offers an opportunity to collect data regularly at scale and at low cost.

From October 2016 to April 2017, we worked with GeoPoll to conduct five survey rounds on MDD-W and find a way to adapt the indicator to SMS. We analysed data from each round, identified gaps and refined the survey instrument. We were able to collect data quickly and identify strengths and weaknesses to make revisions through an iterative process. Through this process, we believe that we have successfully designed an instrument that can be used to monitor MDD-W trends by SMS. Here’s a short summary of what we learned:

1. Using a mix of open-ended and list-based questions helped people better understand our questions.

By using a mix of open-ended and list-based questions, we were able to significantly improve data quality. MDD-W round 1In the first few rounds, we had an unusually high number of respondents who either scored “0” or “10” on the MDD-W score, which are both unlikely under normal circumstances. A score of “0” means that the respondent did not consume food items from any of the 10 core food groups the previous day or night, while a score of “10” means that the respondent consumed food items from all food groups. In the first round, scores of “0” or “10” accounted for 29 percent of all MDD-Wrespondents, but by Round 5 these scores represented only 3 percent of responses. It seems that having respondents reflect about what they ate in the open-ended questions we introduced in later rounds helps them  recall the food items they consumed and answer the subsequent list-based questions more accurately.

2. Keep questions simple.

We originally asked people by SMS whether they ate food items from the core food groups that comprise the MDD-W score. For example, “Yesterday, did you eat any Vitamin A-rich fruits and vegetables such as mangos, carrots, pumpkin, …….” Perhaps respondents thought that they needed to consume food items from both the fruit and vegetable groups in order to reply “yes” to this question. So instead, we split that question into two separate questions (one on Vitamin A-rich fruits and the other on Vitamin A-rich vegetables) to make it easier for the respondent to answer. We did the same for some of the other questions and found a very low percentage of women scoring “0” or “10” on the MDD-W score. Of course there is a trade-off here, and splitting too many questions might lead to a long and unwieldy questionnaire that could frustrate respondents.

3. Let respondents take the survey in their preferred language.

Comprehension remains a challenge in automated surveys, so helping respondents by asking questions in their own language will ensure data quality and limit non-response. In the Malawi study, translating food items into the local language (Chichewa), while keeping the rest of the questionnaire in English, improved comprehension. We recommend providing the respondent with the option to take the survey in their preferred language.

4. Pre-stratify and pre-target to ensure representativeness.

SMS surveys tend to be biased towards people who have mobile phones; we reach a lot of younger, urban men, and relatively few women of reproductive age, our target group for surveys on women’s diet. To ensure we are reaching them, an MDD-W SMS survey should be designed or ‘pre-stratified’ to include a diverse group of respondents. In Malawi, we were able to pre-stratify according to variables that included age, level of education, location and wealth. This allowed us to include women from all walks of life.

5. Post-calibrate to produce estimates that are more comparable to face-to-face surveys.

The MDD-W SMS surveys we conducted produced higher point estimates than those we would expect in face-to-face surveys. This suggests we may wish to consider calibration to adjust for sampling bias, the likely cause for the discrepancy. Calibration is the process of maintaining instrument accuracy by minimizing factors that cause inaccurate measurements. We’re still working on this and hope to find a solution soon. In the meantime, we think we are able to track trends in MDD-W by SMS with some reliability.

 

Postcard from Bangui

Good to be OKING:It may not be new and super large, but the owner claims this phone has a week-long battery life! WFP/Dominique Ferretti

It may not be new and super large, but the owner claims this phone has a week-long battery life!
WFP/Dominique Ferretti

Greetings from the Central African Republic (CAR)! Our team recently visited Bangui and Kaga-Bandoro to help the Country Office team assess how to enhance the current mVAM system and see what other mVAM technologies we might be able to deploy. CAR is a very unique context, because there’s little-to-no cell phone reception outside of main towns. Only 26% of the population own a phone, one of the lowest rates in the world according to the World Bank.  This means that collecting data remotely takes some creativity. The CAR team uses a key informant system, where they contact approximately 200 people around the country each month to collect information on basic commodity prices, market access, population movements, and security issues. The collected information is then shared with the humanitarian community, who appreciate the data, as it’s the only national-level food security data that’s currently collected regularly!

A local woman in Kaga-Bandoro selling a great source of protein and a central African delicacy—caterpillars! WFP/Dominique Ferretti

A local woman in Kaga-Bandoro selling a great source of protein and a central African delicacy—caterpillars!
WFP/Dominique Ferretti

The only downfall to the key informant system is that it doesn’t give us household-level food security information. The CAR team has therefore decided to try a small pilot using household questionnaires in the city of Kaga-Bandoro. Courtesy of UNHAS, we visited the city (more like a very small town!) and the 2 IDP camps it hosts during our day trip. While not that many people had cell phones, enough community members and displaced persons had phones that we’ll be able to get some idea of the food security situation.

Stay tuned for more as the pilot unfolds…!

Chatting with community members as they collect water WFP/Dominique Ferretti

Chatting with community members as they collect water
WFP/Dominique Ferretti

Are you conducting mobile surveys responsibly?

Twitter card data responsibility

The use of mobile technology is a tremendous opportunity to better communicate with people in humanitarian settings. However, these advanced capabilities also involve new privacy and security risks for people in the communities where remote mobile surveys are implemented. We therefore collaborated with the International Data Responsibility Group and Leiden University’s Centre for Innovation to draft a practical guide: ‘Conducting Mobile Surveys Responsibly: A field book for WFP staff’.

The field book outlines the main risks for staff engaged in remote data collection and details best practices for data security, privacy and responsible data approaches in the very complex environments in which WFP operates.

data responsibility front page

If you’re not human then who are you?

Experimenting with chatbots in Nigeria and Haiti

WFP/Lucia Casarin

Testing the bot in Haiti – WFP/Lucia Casarin

Readers of this blog know that the team has been experimenting with chatbots to communicate with disaster-affected communities – read our previous posts about our prototype and the Nielsen Hackathon.

As part of this effort, during recent missions to Haiti and Nigeria, our team went out to talk to communities to find out whether a chatbot would be right for them.

Would a chatbot be a stretch in these communities?

Well it’s not that much of a stretch.

In North East Nigeria, most displaced people live in Maiduguri, a city of over 1 million people. In this ‘urban’ setting connectivity is good, most people own cell phones and many young people use social media and messaging apps. Mobile operators have been offering services that allow people to access the internet by selling ‘social bundles’ (unlimited social media access sold in very small increments) and offer some services for free, including Facebook Light and Facebook Messenger.

In Haiti, three-quarters of the population live in the capital, Port-au-Prince, where 3G connectivity is good and most people use messaging apps to communicate with friends and family. Even in rural and difficult-to-reach communities, leaders and young people own smartphones and connect to the internet. There is a lot of competition between mobile operators so the prices for mobile data are very low. This means that most people can afford to access the internet either via their own smartphone or from shared smartphones.

A

Mobile phones charging station on the road from Léogane Peri to Port-au-Prince WFP/Lucia Casarin

A bare-bones demo

In both countries we tested a simple chatbot that asks people about food prices and what the food security is like in their community. The survey we used was much more basic than our usual mobile questionnaires as we felt it was important to keep things simple at this stage.

For Nigeria, the bot demo was initially in English but we soon translated it into Hausa, the primary language spoken by displaced persons in Maiduguri. In Haiti we made it available both in Creole and French. The chatbot was very responsive on 3G and it even worked with slower 2G connections so the technology works in these contexts. But this was only the starting point, what we really wanted to know was what ‘real’ people thought about the bot.

We organized focus group discussions with displaced people in Maiduguri and with community representatives in Haiti. We helped people access the WFP bot via their Facebook accounts, and they began chatting away.

Sounds cool, but what are the limitations?

Here’s what people said:

First of all, people thought the bot is a convenient, quick, and easy way to get in touch directly with WFP and they really liked that the bot allows them to speak to WFP without intermediaries. They had lot to tell us particularly through the open-ended question where they typed out detailed responses.

In Nigeria, they did tell us that our (somewhat wordy) English-language demo should be translated into Hausa because it would make it easier for everyone to use. Our first group of testers were young people who were already Facebook users and so were familiar with Messenger. It was therefore no surprise that they were interacting smoothly with the bot and able to go through our questionnaire in minutes.

WFP/Jean-Martin Bauer

Testing the bot in Nigeria – WFP/Jean-Martin Bauer

In Haiti, people started interacting with the bot as if it was a human rather than an automated questionnaire so they got stuck pretty fast when it wasn’t as naturally responsive as they’d expected. This means that either we give clearer instructions to people or we add Natural Language Processing capabilities to our bot.

There are of course other barriers. In both countries women appeared to be less likely to own a smartphone. This means that bot users will likely be overwhelmingly young, male and better educated than other people – hardly ‘representative’ of WFP’s target affected population. The free version of the bot is also not always available: in Nigeria only Airtel subscribers can access it, while in Haiti the free service doesn’t exist yet.

This means that the bot would need to be a complement to the other tools we have. We might use data from the bot to obtain a quick situation update, but we will continue relying on other sources for more representative data.