Mind the mode:

Who's texting & who's talking in Malawi?

Malawi mVAM respondent WFP/Alice Clough

Malawi mVAM respondent
WFP/Alice Clough

It’s time for another installment of our Mind the Mode series. For those of you who follow this blog regularly, you know that the mVAM team is continually evaluating the quality of the data we collect. Past Mind the Mode blogs have discussed our work in Mali looking at face-to-face versus voice calls, our comparison of SMS and IVR in Zimbabwe and the differences in the Food Consumption Score (FCS) for face-to-face versus Computer-Assisted Telephone Interviews (CATI) interviews in South Sudan.

This month, we turn our attention to Malawi, where we recently completed a study analyzing the differences in the reduced Coping Strategies Index (rCSI) when it’s collected via CATI and SMS. This indicator helps measure a household’s food security by telling us what actions they might be taking to cope with any stresses such as reducing the number of meals a day or borrowing food or money from friends or family. From February to April 2017, around 2,000 respondents were randomly-selected for an SMS survey and 1,300 respondents were contacted on their mobile phones by an external call centre to complete a CATI survey.

People Profiling: who’s Texting and who’s Talking? 

Across all three rounds, a greater proportion of respondents in both modalities were men who lived in the South and Central Regions of the country and came from male-headed households. However, the respondents taking the SMS survey were much younger (average age 29) than those who took the CATI survey (average age 40). This probably isn’t surprising when you consider that young people across the world tend to be much more interested in new technologies and in Malawi are more likely to be literate.

The results from our mode experiment in Zimbabwe showed that IVR and SMS surveys reached different demographic groups so we figured we might see the same results in Malawi. However, this was surprisingly not the case: both CATI and SMS participants seemed to come from better-off households. In our surveys we determine this by asking them what material the walls of their home are made from (cement, baked bricks, mud, or unbaked bricks).

better off-worse off wall type malawi

More respondents (60%) said they have cement or baked brick walls as opposed to mud or unbaked brick walls, an indicator of being richer.

Digging into the rCSI

So what about the results observed for the rCSI between the two modes? The CATI rCSI distribution shows a peak at zero (meaning that respondents are not employing any negative coping strategies) and is similar to the typical pattern expected of the rCSI in face-to-face surveys (as you can see in the two graphs below).

Density plot for CATI Feb-April 2017

 

SMS rCSI

The SMS results, on the other hand, tend to have a slightly higher rCSI score than in CATI, meaning that respondents to the SMS survey are employing more negative coping strategies than households surveyed via CATI. This is counter-intuitive to what we might expect, especially since the data illustrates that these households are not more vulnerable than CATI respondents. Presumably, they would actually be better educated (read: literate!) to be able to respond to SMS surveys. We’re therefore looking forward to doing some more research in to why this is the case.

Box plot cati rcsi

It’s All in the Numbers

Some interesting patterns in terms of responses were also observed via both modalities. SMS respondents were more likely to respond to all five rCSI questions by entering the same value for each question (think: 00000, 22222…you get the idea!). At the beginning of the survey, SMS respondents were told that they would earn a small airtime credit upon completion of the questionnaire. We conjecture that some respondents may have just entered numbers randomly to complete the questionnaire as quickly as possible and receive their credit. Keep in mind that entering the same value for all five rCSI questions via CATI is a lot more difficult, as the operator is able to ask additional questions to ensure that the respondent clearly understands the question prior to entering the response.  For SMS, there’s no check prohibiting the respondent from dashing through the questionnaire and entering the same response each time.

We also saw that the percentage of respondents stating that they were employing between zero and four strategies was much lower among SMS respondents than CATI respondents across all three months of data collection. Conversely, more respondents (three out of five) in the SMS survey reported that they were using all five negative coping strategies than in the CATI survey. Again, this is counter-intuitive to what we would expect.  It might mean that SMS respondents didn’t always correctly understand the questionnaire or that they didn’t take the time to reflect on each question, completing questions as rapidly as possible to get their credit; or simply entered random numbers in the absence of an operator to validate their responses.  The graphs below illustrate the differences in rCSI responses between CATI and SMS.

Figure 3: Distribution of the number of coping strategies reported by SMS and CATI respondents by months

Figure 3: Distribution of the number of coping strategies reported by SMS and CATI respondents by months

From these results, you can see that we still have a lot to learn on how survey modality affects the results. This is just the start of our research; so expect more to come as the team digs deeper to better understand these important differences.

Postcard from Dakar

mVAM workshop participants all smiles after learning more about IVR WFP/Lucia Casarn

mVAM workshop participants all smiles after learning more about IVR WFP/Lucia Casarn

During the last week of June, staff from WFP HQ’s mVAM team, the West and Central Africa Regional Bureau, and Nigeria and Niger Country Offices met in beautiful Dakar to work together on Interactive Voice Response (IVR) systems for two-way communication. (If you want to dig deep into all details IVR-related, check out the lesson in our mVAM online course!)

We’ve previously blogged about how WFP is responding to the needs of people who have been displaced due to Boko Haram insurgencies in both Nigeria and Niger. When we implemented these operations we also put communication channels in place so beneficiaries are able to contact WFP. In Nigeria, the Maiduguri Field Office created a hotline. Their operators receive an average of 100 calls per day from beneficiaries asking food security-related questions and providing feedback on the operations. The problem is the hotline is only available during working hours and has a limited number of people who can call in at the same time. To work around this they’re therefore looking at how an IVR system can support the call operators who are dealing with high volumes and better manage calls that take place outside of normal office hours. WFP Niger wants to set up a similar hotline system but without full time phone operators. Beneficiaries will call in to an automated IVR system and their queries and feedback recorded by the system and followed up by the Country Office. 

A Nigeria IT Officer working to install a GSM gateway for IVR usage in Maiduguri WFP/Lucia Casarin

A Nigeria IT Officer working to install a GSM gateway for IVR usage in Maiduguri WFP/Lucia Casarin

During the workshop participants were trained by InSTEDD on how to physically deploy IVR using a GSM gateway (a fancy tool that automatically places phone calls) and Verboice, the free open source software they’ve developed to manage these systems. The team also discussed the nitty gritty technical aspects of the system, including creating and modifying call flows (the sequencing of questions), scheduling calls and downloading collected call logs and recordings. Most importantly, participants had the opportunity to share their experiences and challenges with experts in this field and discuss best practices, alternative deployments and technical solutions.

The Country Office staff have now returned to Niger and Nigeria and they’ve already started testing the use of the IVR machines. We’re eager to begin logging data and hearing more from our beneficiaries. So stay tuned!

 

 

 

Postcard from DRC

Congo Call Center operator on the phone with a WFP beneficiary to discuss WFP’s activities

Congo Call Center operator on the phone with a WFP beneficiary to discuss WFP’s activities

Greetings from the Democratic Republic of Congo (DRC)! Two members of the mVAM team recently travelled to Kinshasa to help the WFP Country Office assess how to improve upon its current mVAM system and see what other mVAM technologies we could roll out in the coming months.

mVAM data collection in DRC is conducted nation-wide in collaboration with the Cellule d’Analyses des Indicateurs de Développement, more commonly referred to as CAID. CAID is part of the Congolese National Government, housed within the Prime Minister’s Office, and is responsible for collecting food security and other indicators on a regular basis. Since April 2016, CAID—with technical support from WFP—has been collecting remote food security data across more than 50% of the country. Now this is quite a feat when you consider the vast size of DRC (it would cover most of Western Europe!) coupled with the fact that in many places there is little network coverage.

Map from: http://www.mylifeelsewhere.com/country-size-comparison/belgium/democratic-republic-of-congo

Map from: http://www.mylifeelsewhere.com/country-size-comparison/belgium/democratic-republic-of-congo

During the visit, the mVAM team met with CAID to discuss how to improve its data quality and expand to areas not yet covered by mVAM. This included a visit to the Congo Call Center (CCC). They have a team of operators dedicated to conducting the monthly calls so we went to discuss any challenges they encounter when placing calls. We also brainstormed different ways share the information that CAID collects with the general public. They currently produce a monthly bulletin called ‘m-kengela’ that shares price information and other food security-related details but they also want to share this information with a larger audience. So, together we explored the possibility of creating a Free Basics website that would be accessible to a larger audience. Given the success of our Free Basics pilot in Malawi and the fact that there are two participating mobile network providers within the country, we decided that this would be an ideal way of creating a nation-wide price website. We therefore met with cell phone companies and spent time with CAID mapping out what their site might look like.

Discussions are now underway vis-a-vis the next steps and the mVAM team and CAID are hard at work preparing for the launch of its first Free Basics price website. So stay tuned for more details as rollout takes place over the coming months!

mVAM falls in love in Nepal

WFP/Gaurav Singhal

The mVAM-NeKSAP team (WFP/Gaurav Singhal)

 

The mVAM Team is on the move again. This time our travels took us to Kathmandu, Nepal, where we’re not only excited at the prospect of using mVAM for the first time, but mVAM has also fallen in love and is soon be a proud parent!

The Government of Nepal currently runs a key-informant based food security monitoring system it calls NeKSAP. Each trimester, community leaders in 74 of 75 of Nepal’s Districts gather and use convergence evidence to assess the criticality of the food-security situation in their respective area. This exercise has proven invaluable in directing programming and resources not just for WFP but also for the government and other development/humanitarian organizations across the country. But the process is cumbersome and a bit imprecise when it comes to understanding and responding to a vastly complicated humanitarian landscape (think about the 2015 earthquake in Kathmandu and climate change!).

In 2016, an extreme drought in the high-altitude plateaus of the Mid-Western and Far-Western Development Regions of the country prompted the WFP Nepal Country Office to conduct a face-to-face food security baseline assessment. Given the persistent acute food insecurity in the region the team, collaborating with the Government of Nepal, requested assistance to create a seasonal food security monitoring system, leveraging the agility, efficiency, and cost-effectiveness of mVAM.

This region is so remote that in the past WFP preferred to use airlifts than trucks to deliver assistance. This could not have been a bigger challenge for mVAM given issues with selection bias and phone-ownership. This was compounded by the fact that government insisted on nothing less than high-quality, representative, publishable statistics that could be used in official government figures. If only there was some way we had to reach the most remote and inaccessible regions as well that can only be reached by travelling for several days on foot! As it turns out, the NeKSAP system has provisions for a small network of skilled enumerators to live and work in these regions.

Rural landscape in Bajhang District, located in the North Western region of the country WFP/Bikkil Sthapit

Rural landscape in Bajhang District, located in the North Western region of the country
WFP/Bikkil Sthapit

That is when we had brainwave borrowed from South Asian tradition: an arranged marriage for mVAM (don’t worry, there was a courtship first)…and mNeKSAP was conceived combining the best of traditional face-to-face assessment with mVAM! Why only rely on one survey mode?  For individuals without phones we decided to use the NeKSAP enumerators to do traditional face-to-face assessments.

Furthermore, all the individuals were first interviewed in a face-to-face pre-winter baseline. This means that not only is the data representative, ensuring coverage of non-phone owners, mNeKSAP also provides a rare panel data set, re-interviewing the same individuals every trimester over a year.  Panel data is the gold-standard for doing causal inference. There is much more work to be done of course but we’ll keep you updated on this exciting new collaboration.

VAM Talks Episode 11: IDPs in Nigeria test our chatbot

Episode 11:   20 June 2017

Logo2Jean-Martin Bauer and Seokjin Han travel to North-Eastern Nigeria to test out our humanitarian chatbot with IDPs affected by the Boko Haram insurgency,

After the rains: sprouts of green and mVAM in South Sudan

Our South Sudan mVAM operators at work (WFP/Angie Lee)

Our South Sudan mVAM operators at work (WFP/Angie Lee)

Greetings from an ever-green Juba! The last time we reported from South Sudan it was dry and dusty everywhere. This time our visit coincided with the start of the rainy season – a welcome respite to the scorching heat that lasted for months.

Other than the heat, there are many challenges in South Sudan, particularly when trying to set up an mVAM system. South Sudan is one of the worst ranking countries in terms of mobile phone penetration and connectivity: according to 2016 ITU data, approximately 24 percent of the population have mobile cellular subscriptions and merely 4.3 percent of households own a computer. The ongoing conflict has only made the situation worse. We found out that network coverage has significantly deteriorated since mVAM activities first started in February 2016. A case in point: one major network operator, which reportedly had the largest outreach in the country, reduced its coverage from 70 to 15 percent. Our mVAM operators told us that completing a 10-minute survey with one single phone call was nearly impossible, because the line is constantly dropping.

Even when a call does go through, it is extremely difficult to pinpoint the respondent’s location. People are on the move fleeing the conflict (more than 950,000 South Sudanese have crossed the border into Uganda alone according to the latest UNHCR estimates) and phone numbers keep changing (the average shelf life of a SIM is short as people are on the move and network coverage varies greatly between different areas). To make things even more complicated, the administrative boundaries of the country are also shifting (in addition to the existing 10 states, an additional 22 states have been newly created).

Being mindful of these challenges, we had previously recommended that the country office start contacting a pool of key informants who are easier to reach and were able to collect data on markets, displacement, and road access in the Greater Upper Nile Region. However, even here we are confronted with the challenge of collecting data in a highly politically-divided context. Relying exclusively on key informant sources can give you a biased picture of the situation on the ground, especially where the informants speak for specific interest groups. It is therefore necessary to triangulate various sources of key informant information and complement them with other secondary or even primary household data when possible.

Does all of this mean that there is no future for mVAM in South Sudan? On the contrary, we found that the demand for mobile surveys is there both for WFP and the humanitarian community at large. After all, South Sudan is a complex emergency where ‘putting boots on the ground’ is often not possible and we need all the creativity and tools we can muster. In fact, WFP South Sudan has been conducting mobile surveys for market monitoring and rapid emergency food security assessments (the latest one took place in select famine-affected counties). Similarly, other NGO and development partners on the ground are also conducting mobile surveys for programme or food security monitoring.

Moving forward, we have identified, together with the South Sudan VAM team, two areas of opportunity where we can scale mVAM: i) urban food security monitoring in selected hotspots and interest points and ii) complementing the early warning bulletin jointly produced by the Ministry of Humanitarian Affairs and Disaster Management and WFP with mVAM key informant data.

Green fellows like these are often found at WFP premises (WFP/Marie Enlund)

Green fellows like these are often found at WFP premises (WFP/Marie Enlund)

Mind the mode …. and the non-response

How voice and face-to-face survey data compares in Mali

This is the third entry in our ‘Mind the Mode’ series on the mVAM blog. We are constantly assessing our data collection modalities to better understand what produces the most-accurate results and what biases may be present. One of our recent experiments took us to Mali, where we were comparing the food consumption score between face-to-face (F2F) interviews versus mVAM live calls.

It’s all in the details
To do this, in February and March, the WFP team first conducted a baseline assessment in four regions of the country. As part of the baseline, we collected phone numbers from participants. Approximately 7-10 days later, we then re-contacted those households who had phones, reaching roughly half of those encountered during the face-to-face survey. We weren’t able to contact the other households. To ensure the validity of the results, we made sure the questionnaire was the exact same between the F2F and telephone interviews. Any differences in wording or changes in the way in which the questions were asked could adversely affect our analysis.

The findings from our analysis were quite interesting. We found that food consumption scores (FCS) collected via the mVAM survey tended to be slightly higher than those collected via the face-to-face survey. The graph below illustrates this shift to higher scores between the two rounds. Higher FCS via mVAM versus F2F surveys is not atypical to Mali. We’ve observed similar outcomes in South Sudan and other countries where mVAM studies have taken place.

mali dist

 

Why could this be? There are two main reasons that could explain this difference. Either it might be due to the data collection modality (i.e., people report higher food consumption scores on the phone)? Or, a perhaps a selection bias is occurring? Remember that we were only able to contact roughly half of the participants from the F2F survey during the telephone calls. So, it’s possible that people who responded to the phone calls are less food insecure, which could make sense, since we often see that the poorest of the poor either don’t own a phone or have limited economic means to charge their phone or purchase phone credit.

To test these hypotheses, we dug a bit deeper.

Same same…
Are people telling the same story on the phone versus face-to-face? Based on our results, the answer is yes! If we compare the same pool of respondents who participated in both the F2F and telephone survey rounds, their food security indicators are more or less the same. For example, the mean mVAM FCS was 56.21 while the mean F2F FCS was 55.65, with no statistically significant difference between the two.

But different…
So what about selection bias? In the F2F round, there are essentially three groups of people: 1) those who own phones and participated in both the F2F and mVAM survey; 2) people who own phones but didn’t participate in the mVAM survey, because they either didn’t answer the calls or their phone was off; and 3) people who do not own a phone and thus couldn’t participate in the mVAM survey.

People who replied to the mVAM survey have overall higher FCS than those that we were unable to contact. What we learned from this experiment is that bias does not only come from the households that do not own a phone but also from non-respondents (those households who shared their phone number and gave consent but then were not reachable later on for the phone interview). Possible reasons why they were not reachable could be that they have less access to electricity to charge their phone or that they live in areas with bad network coverage. The graph below illustrates the distribution by respondent type and their respective FCS.

mali boxp

When you compare the demographics of people in these three groups based on the data collected in the baseline, you can see that there are significant differences, as per the example below. Notice that the education levels of respondents varies amongst the three groups—those without a phone tend to be less educated than those who own a phone and participated in the mVAM survey.

mali profile

This study taught us a valuable lesson. While we are confident that there is no statistically significant difference between face-to-face and phone responses within the Mali context, there is a selection bias in mVAM-collected data. By not including those without phones as well as those who did not respond, we are missing an important (and likely poorer) subset of the population, meaning that the reported FCS is likely higher than it may be if these groups were included. One way to account for this bias is to ensure that telephone operators attempt to contact the households numerous times, over the course of several days. It’s important that they really try to reach them. The team is also studying how to account for this bias in our data analyses.

Trial and Error: How we found a way to monitor nutrition through SMS in Malawi

WFP/Alice Clough

WFP/Alice Clough

Over the last ten months we have been testing if we can use mobile phones to collect nutrition indicators. One of these experiments involved using SMS to ask questions about women’s diet quality via the Minimum Dietary Diversity – Women (MDD-W) indicator.  The MDD-W involves asking simple questions about whether women of reproductive age (15-49 years) consumed at least five out of ten defined food groups. We were interested in using SMS surveys to measure MDD-W, because SMS offers an opportunity to collect data regularly at scale and at low cost.

From October 2016 to April 2017, we worked with GeoPoll to conduct five survey rounds on MDD-W and find a way to adapt the indicator to SMS. We analysed data from each round, identified gaps and refined the survey instrument. We were able to collect data quickly and identify strengths and weaknesses to make revisions through an iterative process. Through this process, we believe that we have successfully designed an instrument that can be used to monitor MDD-W trends by SMS. Here’s a short summary of what we learned:

1. Using a mix of open-ended and list-based questions helped people better understand our questions.

By using a mix of open-ended and list-based questions, we were able to significantly improve data quality. MDD-W round 1In the first few rounds, we had an unusually high number of respondents who either scored “0” or “10” on the MDD-W score, which are both unlikely under normal circumstances. A score of “0” means that the respondent did not consume food items from any of the 10 core food groups the previous day or night, while a score of “10” means that the respondent consumed food items from all food groups. In the first round, scores of “0” or “10” accounted for 29 percent of all MDD-Wrespondents, but by Round 5 these scores represented only 3 percent of responses. It seems that having respondents reflect about what they ate in the open-ended questions we introduced in later rounds helps them  recall the food items they consumed and answer the subsequent list-based questions more accurately.

2. Keep questions simple.

We originally asked people by SMS whether they ate food items from the core food groups that comprise the MDD-W score. For example, “Yesterday, did you eat any Vitamin A-rich fruits and vegetables such as mangos, carrots, pumpkin, …….” Perhaps respondents thought that they needed to consume food items from both the fruit and vegetable groups in order to reply “yes” to this question. So instead, we split that question into two separate questions (one on Vitamin A-rich fruits and the other on Vitamin A-rich vegetables) to make it easier for the respondent to answer. We did the same for some of the other questions and found a very low percentage of women scoring “0” or “10” on the MDD-W score. Of course there is a trade-off here, and splitting too many questions might lead to a long and unwieldy questionnaire that could frustrate respondents.

3. Let respondents take the survey in their preferred language.

Comprehension remains a challenge in automated surveys, so helping respondents by asking questions in their own language will ensure data quality and limit non-response. In the Malawi study, translating food items into the local language (Chichewa), while keeping the rest of the questionnaire in English, improved comprehension. We recommend providing the respondent with the option to take the survey in their preferred language.

4. Pre-stratify and pre-target to ensure representativeness.

SMS surveys tend to be biased towards people who have mobile phones; we reach a lot of younger, urban men, and relatively few women of reproductive age, our target group for surveys on women’s diet. To ensure we are reaching them, an MDD-W SMS survey should be designed or ‘pre-stratified’ to include a diverse group of respondents. In Malawi, we were able to pre-stratify according to variables that included age, level of education, location and wealth. This allowed us to include women from all walks of life.

5. Post-calibrate to produce estimates that are more comparable to face-to-face surveys.

The MDD-W SMS surveys we conducted produced higher point estimates than those we would expect in face-to-face surveys. This suggests we may wish to consider calibration to adjust for sampling bias, the likely cause for the discrepancy. Calibration is the process of maintaining instrument accuracy by minimizing factors that cause inaccurate measurements. We’re still working on this and hope to find a solution soon. In the meantime, we think we are able to track trends in MDD-W by SMS with some reliability.

 

Postcard from Bangui

Good to be OKING:It may not be new and super large, but the owner claims this phone has a week-long battery life! WFP/Dominique Ferretti

It may not be new and super large, but the owner claims this phone has a week-long battery life!
WFP/Dominique Ferretti

Greetings from the Central African Republic (CAR)! Our team recently visited Bangui and Kaga-Bandoro to help the Country Office team assess how to enhance the current mVAM system and see what other mVAM technologies we might be able to deploy. CAR is a very unique context, because there’s little-to-no cell phone reception outside of main towns. Only 26% of the population own a phone, one of the lowest rates in the world according to the World Bank.  This means that collecting data remotely takes some creativity. The CAR team uses a key informant system, where they contact approximately 200 people around the country each month to collect information on basic commodity prices, market access, population movements, and security issues. The collected information is then shared with the humanitarian community, who appreciate the data, as it’s the only national-level food security data that’s currently collected regularly!

A local woman in Kaga-Bandoro selling a great source of protein and a central African delicacy—caterpillars! WFP/Dominique Ferretti

A local woman in Kaga-Bandoro selling a great source of protein and a central African delicacy—caterpillars!
WFP/Dominique Ferretti

The only downfall to the key informant system is that it doesn’t give us household-level food security information. The CAR team has therefore decided to try a small pilot using household questionnaires in the city of Kaga-Bandoro. Courtesy of UNHAS, we visited the city (more like a very small town!) and the 2 IDP camps it hosts during our day trip. While not that many people had cell phones, enough community members and displaced persons had phones that we’ll be able to get some idea of the food security situation.

Stay tuned for more as the pilot unfolds…!

Chatting with community members as they collect water WFP/Dominique Ferretti

Chatting with community members as they collect water
WFP/Dominique Ferretti

Are you conducting mobile surveys responsibly?

Twitter card data responsibility

The use of mobile technology is a tremendous opportunity to better communicate with people in humanitarian settings. However, these advanced capabilities also involve new privacy and security risks for people in the communities where remote mobile surveys are implemented. We therefore collaborated with the International Data Responsibility Group and Leiden University’s Centre for Innovation to draft a practical guide: ‘Conducting Mobile Surveys Responsibly: A field book for WFP staff’.

The field book outlines the main risks for staff engaged in remote data collection and details best practices for data security, privacy and responsible data approaches in the very complex environments in which WFP operates.

data responsibility front page