New places, new tools: what’s up next for mVAM?

KOICA pic 2

We’ve just got back from Rwanda where we were holding a workshop on using mVAM to expand real-time food security and nutrition monitoring with Internally Displaced Persons (IDPs) and refugee populations. The project, which is made possible by the support of the Korean International Cooperation Agency (KOICA), will be implemented in ten countries in sub-Saharan Africa where WFP works.

What’s the project?

The KOICA project has two aims. First, it aims to empower information exchange with marginalized populations, specifically IDPs and Refugees. Secondly, it supports the collection of food security and nutrition data using the latest mobile and satellite technologies. This will happen in ten countries in Sub-Saharan Africa: the Central African Republic (CAR),The Democratic Republic of Congo (DRC), Kenya, Malawi, Niger, Nigeria, Rwanda, Somalia, South Sudan and Uganda.

How are we going to do this?

As you know, two-way communication systems are an important part of our work. As well as getting information that we can use to inform WFP programmes, we want to ensure that the line is open so that people in the communities we serve can contact us and access information that is useful to them. We’ve already been using Interactive Voice Response and live calls to share information with affected populations, and are now expanding our toolbox to include new technologies: Free Basics and a chatbot.

Remote data collection isn’t just done by mobile phones – VAM already uses other sources, such as  satellite imagery analysis – to understand the food security situation on the ground.  Under this project, we’ll also help countries incorporate similar analysis which will complement two-way communication systems to provide a fuller picture of the food security situation.

Finally, we’re going to harness our knowledge of Call Detail Records analysis: de-identified metadata collected via cell phone towers about the number of calls or messages people are sending and which towers they are using. We have already used this technique in Haiti to track displacement after Hurricane Matthew, and we’re really excited to transfer these ideas to another context to ensure we get up-to-date information on where affected communities are so we can better target food assistance in the right locations.

What happened at the workshop?

Representatives from all 10 country offices, three regional bureaus and staff from HQ came together to discuss the three main project components. During the workshop, the different country offices had the chance to learn more from members of the mVAM team about the specific tools they can harness and ensure their collected data is high quality, standardised and communicated effectively. However, the best part about bringing everyone together was that country teams could share their experiences about how they are already using mVAM tools. We heard from the Malawi country office about their Free Basics pilot, and Niger and Nigeria explained how they’re implementing IVR so affected communities can easily contact WFP, even after work hours. Sharing their different experiences and learning about how different tools have worked in each context not only gave everyone an overview of what mVAM is doing so far, it also helped everyone understand the implementation challenges and how to overcome them.

What’s next for the KOICA project?

We’re really excited for the next stage of the project. Each country office has now planned what tools they’re going to use to increase their communications with affected communities and how they will improve their existing data collection systems. It’s going to be great to see the impact these tools will have not only on WFP’s response, but also how they will empower the communities we’re serving. 

Mind the mode:

Who's texting & who's talking in Malawi?

Malawi mVAM respondent WFP/Alice Clough

Malawi mVAM respondent
WFP/Alice Clough

It’s time for another installment of our Mind the Mode series. For those of you who follow this blog regularly, you know that the mVAM team is continually evaluating the quality of the data we collect. Past Mind the Mode blogs have discussed our work in Mali looking at face-to-face versus voice calls, our comparison of SMS and IVR in Zimbabwe and the differences in the Food Consumption Score (FCS) for face-to-face versus Computer-Assisted Telephone Interviews (CATI) interviews in South Sudan.

This month, we turn our attention to Malawi, where we recently completed a study analyzing the differences in the reduced Coping Strategies Index (rCSI) when it’s collected via CATI and SMS. This indicator helps measure a household’s food security by telling us what actions they might be taking to cope with any stresses such as reducing the number of meals a day or borrowing food or money from friends or family. From February to April 2017, around 2,000 respondents were randomly-selected for an SMS survey and 1,300 respondents were contacted on their mobile phones by an external call centre to complete a CATI survey.

People Profiling: who’s Texting and who’s Talking? 

Across all three rounds, a greater proportion of respondents in both modalities were men who lived in the South and Central Regions of the country and came from male-headed households. However, the respondents taking the SMS survey were much younger (average age 29) than those who took the CATI survey (average age 40). This probably isn’t surprising when you consider that young people across the world tend to be much more interested in new technologies and in Malawi are more likely to be literate.

The results from our mode experiment in Zimbabwe showed that IVR and SMS surveys reached different demographic groups so we figured we might see the same results in Malawi. However, this was surprisingly not the case: both CATI and SMS participants seemed to come from better-off households. In our surveys we determine this by asking them what material the walls of their home are made from (cement, baked bricks, mud, or unbaked bricks).

better off-worse off wall type malawi

More respondents (60%) said they have cement or baked brick walls as opposed to mud or unbaked brick walls, an indicator of being richer.

Digging into the rCSI

So what about the results observed for the rCSI between the two modes? The CATI rCSI distribution shows a peak at zero (meaning that respondents are not employing any negative coping strategies) and is similar to the typical pattern expected of the rCSI in face-to-face surveys (as you can see in the two graphs below).

Density plot for CATI Feb-April 2017

 

SMS rCSI

The SMS results, on the other hand, tend to have a slightly higher rCSI score than in CATI, meaning that respondents to the SMS survey are employing more negative coping strategies than households surveyed via CATI. This is counter-intuitive to what we might expect, especially since the data illustrates that these households are not more vulnerable than CATI respondents. Presumably, they would actually be better educated (read: literate!) to be able to respond to SMS surveys. We’re therefore looking forward to doing some more research in to why this is the case.

Box plot cati rcsi

It’s All in the Numbers

Some interesting patterns in terms of responses were also observed via both modalities. SMS respondents were more likely to respond to all five rCSI questions by entering the same value for each question (think: 00000, 22222…you get the idea!). At the beginning of the survey, SMS respondents were told that they would earn a small airtime credit upon completion of the questionnaire. We conjecture that some respondents may have just entered numbers randomly to complete the questionnaire as quickly as possible and receive their credit. Keep in mind that entering the same value for all five rCSI questions via CATI is a lot more difficult, as the operator is able to ask additional questions to ensure that the respondent clearly understands the question prior to entering the response.  For SMS, there’s no check prohibiting the respondent from dashing through the questionnaire and entering the same response each time.

We also saw that the percentage of respondents stating that they were employing between zero and four strategies was much lower among SMS respondents than CATI respondents across all three months of data collection. Conversely, more respondents (three out of five) in the SMS survey reported that they were using all five negative coping strategies than in the CATI survey. Again, this is counter-intuitive to what we would expect.  It might mean that SMS respondents didn’t always correctly understand the questionnaire or that they didn’t take the time to reflect on each question, completing questions as rapidly as possible to get their credit; or simply entered random numbers in the absence of an operator to validate their responses.  The graphs below illustrate the differences in rCSI responses between CATI and SMS.

Figure 3: Distribution of the number of coping strategies reported by SMS and CATI respondents by months

Figure 3: Distribution of the number of coping strategies reported by SMS and CATI respondents by months

From these results, you can see that we still have a lot to learn on how survey modality affects the results. This is just the start of our research; so expect more to come as the team digs deeper to better understand these important differences.

Trial and Error: How we found a way to monitor nutrition through SMS in Malawi

WFP/Alice Clough

WFP/Alice Clough

Over the last ten months we have been testing if we can use mobile phones to collect nutrition indicators. One of these experiments involved using SMS to ask questions about women’s diet quality via the Minimum Dietary Diversity – Women (MDD-W) indicator.  The MDD-W involves asking simple questions about whether women of reproductive age (15-49 years) consumed at least five out of ten defined food groups. We were interested in using SMS surveys to measure MDD-W, because SMS offers an opportunity to collect data regularly at scale and at low cost.

From October 2016 to April 2017, we worked with GeoPoll to conduct five survey rounds on MDD-W and find a way to adapt the indicator to SMS. We analysed data from each round, identified gaps and refined the survey instrument. We were able to collect data quickly and identify strengths and weaknesses to make revisions through an iterative process. Through this process, we believe that we have successfully designed an instrument that can be used to monitor MDD-W trends by SMS. Here’s a short summary of what we learned:

1. Using a mix of open-ended and list-based questions helped people better understand our questions.

By using a mix of open-ended and list-based questions, we were able to significantly improve data quality. MDD-W round 1In the first few rounds, we had an unusually high number of respondents who either scored “0” or “10” on the MDD-W score, which are both unlikely under normal circumstances. A score of “0” means that the respondent did not consume food items from any of the 10 core food groups the previous day or night, while a score of “10” means that the respondent consumed food items from all food groups. In the first round, scores of “0” or “10” accounted for 29 percent of all MDD-Wrespondents, but by Round 5 these scores represented only 3 percent of responses. It seems that having respondents reflect about what they ate in the open-ended questions we introduced in later rounds helps them  recall the food items they consumed and answer the subsequent list-based questions more accurately.

2. Keep questions simple.

We originally asked people by SMS whether they ate food items from the core food groups that comprise the MDD-W score. For example, “Yesterday, did you eat any Vitamin A-rich fruits and vegetables such as mangos, carrots, pumpkin, …….” Perhaps respondents thought that they needed to consume food items from both the fruit and vegetable groups in order to reply “yes” to this question. So instead, we split that question into two separate questions (one on Vitamin A-rich fruits and the other on Vitamin A-rich vegetables) to make it easier for the respondent to answer. We did the same for some of the other questions and found a very low percentage of women scoring “0” or “10” on the MDD-W score. Of course there is a trade-off here, and splitting too many questions might lead to a long and unwieldy questionnaire that could frustrate respondents.

3. Let respondents take the survey in their preferred language.

Comprehension remains a challenge in automated surveys, so helping respondents by asking questions in their own language will ensure data quality and limit non-response. In the Malawi study, translating food items into the local language (Chichewa), while keeping the rest of the questionnaire in English, improved comprehension. We recommend providing the respondent with the option to take the survey in their preferred language.

4. Pre-stratify and pre-target to ensure representativeness.

SMS surveys tend to be biased towards people who have mobile phones; we reach a lot of younger, urban men, and relatively few women of reproductive age, our target group for surveys on women’s diet. To ensure we are reaching them, an MDD-W SMS survey should be designed or ‘pre-stratified’ to include a diverse group of respondents. In Malawi, we were able to pre-stratify according to variables that included age, level of education, location and wealth. This allowed us to include women from all walks of life.

5. Post-calibrate to produce estimates that are more comparable to face-to-face surveys.

The MDD-W SMS surveys we conducted produced higher point estimates than those we would expect in face-to-face surveys. This suggests we may wish to consider calibration to adjust for sampling bias, the likely cause for the discrepancy. Calibration is the process of maintaining instrument accuracy by minimizing factors that cause inaccurate measurements. We’re still working on this and hope to find a solution soon. In the meantime, we think we are able to track trends in MDD-W by SMS with some reliability.

 

What we found at the market: using Free Basics in Malawi

FreeBasicsAd_Chichewa

We wrote to you back in November about one of our new innovations – our Free Basics website ‘Za Pamsika’ where we’re posting commodity prices using the weekly price data we’re collecting through our mVAM operators on a free website. We said that the project had the potential to reach millions of Malawians – well, a lot has happened since then.

Rather than continuing to willfully upload prices while watching our user statistics go up and down, we went to Malawi to carry out a short ground truth study and get some first hand user feedback.  The aim of the mission was to investigate the best way of using the website and interrogate the assumptions we’d made when designing it.

With this in mind, we tried to answer two big questions:

  1. Who can access our website – what are the potential barriers and how can we work around them?
  2. Do Malawians really want a website where they can find out maize and beans prices?

So we went to rural and urban markets in the Central and Southern regions to speak to the mVAM traders and the consumers in their markets about their mobile phone usage and market activity and to get their feedback on the website.

What kind of answers did we get?

First – access issues. While you don’t need a smartphone to access the website we knew that mobile penetration in Malawi is low. So we were most worried about the prevalence of internet-enabled phones and network coverage. From our study we found out that while we aren’t going to be able to reach everyone in Malawi via a website, we can still communicate with people. Network coverage was a problem in some areas. However, overall we found that most of the traders had internet enabled phones or wanted to buy one. We also found that Malawi’s MNOs have been recently trying to push out better network coverage. All good news for future reach of the website.

Actually the biggest barrier was language and literacy. While English is the national language of Malawi, most of the literate people we spoke to were much more comfortable reading and writing in Chichewa because that’s what they were taught in. While they were very enthusiastic about the website content when it was explained to them, they found the initial design (all in English and text heavy) confusing and difficult to use. Luckily this is an easy change to make so we did a quick redesign of the website and translated it into Chichewa:

malawiblog1

With our new design we headed back into the markets and got much better feedback. Rather than just saying that they liked the website content they could really interact with it and were making comments on the different maize and beans prices.

The second barrier we found was digital literacy. Many of the people we spoke to had internet-enabled phones but either didn’t know how to use them or didn’t even realise that they had the internet on them! Unlike the language change this is not a quick fix. This was particularly prominent amongst the women we spoke to, none of whom were comfortable with mobile internet. We’re therefore going to partner with civil society organisations promoting digital literacy. WFP has a network of partners and farmers on the ground who they reach out to with climate information so we’re going to try and use these focal points to communicate our prices with vulnerable populations and communities who have limited access to information.

IMG_1205

But do Malawians really want a ‘Za Pamsika’ website?

It turned out that maize and beans prices really are something that people want to see on the website. The recent drought was on everyone’s minds and they were really emphasising how much of a difference getting a good price could make. People were also already using their phones to get prices – by calling their friends or other traders in different areas and were quite enthusiastic about the possibility of getting this information for free.

With these learnings in mind and feeling confident with our website redesign and excited to be working closely with the country office, we embarked on our next steps. We now have a new focal point in the Lilongwe office who’s looking after the project and in a much better placed position than us in Rome to reach out to millions of Malawians. By this point over 25,000 people had already visited the Za Pamsika website but we knew our reach could be much further. We therefore started experimenting with ways of advertising the website.

malawiblog2

First – we decided to take out a Facebook ad to try and raise the site profile so we created our own ‘Za Pamsika’ page on Facebook and put out some ads in English and Chichewa. We were pretty excited when they started showing up on Malawian colleagues’ Facebook newsfeeds and within 10 days we’d reached more than 130,000 people and got 650 likes to our Facebook page.

What we didn’t expect was the organic reaction we’d get to our page. Within 3 days we’d not only reached more than 80,000 people with our post, we’d also seen that people started having conversations about maize prices on our advert.  People have also started messaging us about whether we can add their market to our website. We’re also getting comments about what other commodities we should add, for instance more seasonal foods such as groundnuts or soya. Most excitingly we even had someone knock on the door of the sub-office to inquire about the website after seeing our advert!

On a second mission in April we went out to the markets in Lilongwe again armed with our new ‘Za Pamsika’ posters. We were putting them up in the trader’s shops and were pretty quickly swamped with people excited about the website and how it could save them money. But again – everyone was asking us to add more food prices to the site – it seems like Malawians just keep wanting to know more about ‘things you find in the market’!

IMG_1201

So what’s next for Za Pamsika?

We’ve got our new focal point Khataza on board who’s taking charge of the website. First up, taking requests into account, we will be adding other seasonal commodities to the website. We’re going to continue experimenting with our Facebook ads and start using our Facebook page to reach out and engage with people about what they’d like on the page. We’ve also got some new partnerships coming up with civil society organisations who are keen to spread the word about ‘Za Pamsika’ and who we can work with to break down access barriers to this information.

Are millions in Malawi being reached? Not yet – but we’re getting there.

Can we reach rural women via mobile phone? Kenya case study

SONY DSC

WFP/Kusum Hachhethu

 

A few months ago, we published a blog post on our plans to test collecting nutrition data through SMS in Malawi and through live voice calls in Kenya. We just got back from Kenya where we conducted a large-scale mode experiment with ICRAF to compare nutrition data collected face-to-face with data collected through phone calls placed by operators at a call center. But before we started our experiment, we did a qualitative formative study to understand rural women’s phone access and use.

We traveled to 16 villages in Baringo and Kitui counties in Kenya, where we conducted focus group discussions and in-depth interviews with women. We also conducted key informant interviews with mobile phone vendors, local nutritionists, and local government leaders.

So in Kenya, can rural women be reached via mobile phone?

Here are the preliminary findings from our qualitative study:

  1. Ownership: Women’s phone ownership is high in both counties. However, ownership was higher in Kitui than Baringo, which is more pastoralist. From our focus group discussions and interviews, we estimate that 80-90% of women own phones in Kitui and 60-70% own phones in Baringo.
  1. Access: The majority of women had access to phones through inter- and intra-household sharing even if they didn’t own one themselves. This suggests that even women who don’t own a phone personally have access to phones that they may be able to use to participate in phone surveys.
  1. Usage: Women mostly use phones to make and receive calls, not send SMS. This supports our hypothesis that voice calls, not SMS, would be the optimal modality to reach women through mobile surveys.
  1. Willingness: Women were enthusiastic about participating in phone surveys during our focus group discussions and in-depth interviews, implying that they are interested in phone surveys and willing to take part.
  1. Trust: Unknown numbers create trust issues, but they are not insurmountable. Women voiced concerns about receiving phone calls from unknown numbers. Despite these trust issues, we were eventually able to successfully conduct our phone surveys after sensitizing the community, using existing community and government administration structures.
  1. Network: Poor network coverage, not gender norms or access, is the biggest barrier to phone surveys in the two counties. Women identified network coverage as the biggest barrier for communication. Some parts of the counties had poor to no network coverage. However, we found that phone ownership was high even in these areas, and women would travel to find network hotspots to make or receive phone calls.

So in conclusion, yes, in Kenya it is possible to reach rural women by phone.
Our findings from Kitui and Baringo counties show that we can reach women in similar contexts with mobile methodologies to collect information on their diet as well as their child’s diet.

We are also analysing the quantitative data from our mode experiment to find out whether data on women and children’s diet collected via live phone operators gives the same results as data collected via traditional face-to-face interviews.

The El Niño Aftermath: Tracking Hunger in the Millions in Southern Africa

We’ve been writing a lot about how mVAM can help in conflict situations where whole areas are cut off because of violence or an epidemic (see our blogs on Yemen, Somalia, Iraq and article on Ebola). But over the past year, the world was disrupted by another type of event- a climatic one: El Niño. The El Niño weather pattern results from a warming of sea temperatures in the Pacific roughly every three to seven years. This El Niño was one of the strongest on record.  The reason why El Niño was so concerning is its global reach, it didn’t just affect the Pacific; places as far away as Guatemala, Pakistan, Indonesia and Ethiopia were all at risk of floods and/or droughts. While the El Niño itself has abated, it has left millions hungry in its wake (current estimates are that 60 million people are food insecure globally). And a La Niña year is looming.

One area that has been particularly affected is Southern Africa. Across the region, this year’s rainfall season was the driest in the last 35 years. Most farmers are facing significantly reduced and delayed harvests.

El Niño hit when Southern Africa was already vulnerable to food insecurity. The region had already experienced a poor 2014-15 harvest season, meaning that food stocks were already depleted. Now, after El Niño, roughly 41 million people are classified as food insecure. On 13 June 2016 WFP categorized the region as an L3 emergency – a situation requiring the highest level of humanitarian support. We’re therefore dramatically expanding our national food security monitoring in the region so WFP can quickly provide as much relevant food security information as possible to effectively respond to the crisis.

Predictions that this El Niño would have a big effect had already started coming in 2015 so we began setting up mobile monitoring in countries that were particularly vulnerable to El Niño. We started in Malawi which had very disruptive weather patterns looming (potentially too much rain in the north and huge rainfall deficits in the south). We lacked current household data to track the impact on food security across the country.

To get information quickly and cheaply, we started a monthly SMS survey with GeoPoll in December 2015. And Malawians sure were quick to respond! In 24 hours, we had 1,000 questionnaires completed.  When analyzing the results, we wanted to make sure people were understanding our texts. The adult literacy rate in Malawi is only 61.3% so we kept the questionnaire short and as simple as possible. We included questions for one food security indicator- the reduced coping strategy index (rCSI) which asks people about the coping strategies they are using when they don’t have enough to eat. We also checked that the data made sense, and in general, the rCSI behaved as we would suspect. It was correlated with people’s messages about their community’s food security situation and their wealth status. As with all of our surveys, we are continually improving them. In this case, we increased our sample size and district quotas to capture more people in rural areas.

Monitoring Maize Prices

IMG_0095Market prices, especially maize prices, are key to Malawians’ food security. Maize is the staple food, used to make nsima which is consumed daily. So to monitor market prices in 17 hotspot districts, we collected phone numbers from over 100 traders in 51 markets throughout Malawi. We first tried asking them prices by text message, but we didn’t receive many responses.  It seems like sending back a series of texts is a bit too much to ask of traders who volunteered out of their own good will to participate in our market survey. We therefore set up a small call center in WFP’s country office. We trained two operators, and they were quickly placing calls to traders every week. When they could just answer a quick phone call instead of having to type in answers, traders willingly reported current commodity prices.

Our latest report from June 2016 shows that maize prices are now between 50 and 100 percent higher than this time last year. This is having a big effect on Malawians. As you can see from our word cloud, alarmingly ‘not-enough’ featured prominently in our open ended question about maize.

word cloud_cropped

Nutrition Surveillance for the first time

In most countries, we have been concentrating on household level indicators like food consumption. But health centers treating malnutrition could potentially give us important indications of the nutrition situation of different parts of the country. In Malawi, WFP works with health centers to address moderate acute malnutrition (MAM) in Malawi by providing fortified blended foods. So to make the most of our call center, we decided to call these health centers every two weeks and track malnutrition admission data for children (aged 6-59 months) and for adults with HIV/AIDS or tuberculosis. In the first six weeks of monitoring, we saw a big increase in the number of moderate acute malnutrition admissions for children increased greatly where severe acute malnutrition rates did not show a clear pattern. We dug further, and the Ministry of Health had initiated mass screenings to enroll malnourished children in nutrition programmes which generally pick up moderately malnourished children. With health center admission data, it’s important to check what else is going on in the country. We’re hoping to soon pilot contacting mothers of malnourished children about their children’s progress to gain additional insight into the nutrition status of vulnerable populations in Malawi.

Now that we have Malawi firmly established, we’ve started reporting on Madagascar and our data collection is ongoing in Zambia, Lesotho and Mozambique. So watch this space for more news about how we get on in these next few months.