Trial and Error: How we found a way to monitor nutrition through SMS in Malawi

WFP/Alice Clough

WFP/Alice Clough

Over the last ten months we have been testing if we can use mobile phones to collect nutrition indicators. One of these experiments involved using SMS to ask questions about women’s diet quality via the Minimum Dietary Diversity – Women (MDD-W) indicator.  The MDD-W involves asking simple questions about whether women of reproductive age (15-49 years) consumed at least five out of ten defined food groups. We were interested in using SMS surveys to measure MDD-W, because SMS offers an opportunity to collect data regularly at scale and at low cost.

From October 2016 to April 2017, we worked with GeoPoll to conduct five survey rounds on MDD-W and find a way to adapt the indicator to SMS. We analysed data from each round, identified gaps and refined the survey instrument. We were able to collect data quickly and identify strengths and weaknesses to make revisions through an iterative process. Through this process, we believe that we have successfully designed an instrument that can be used to monitor MDD-W trends by SMS. Here’s a short summary of what we learned:

1. Using a mix of open-ended and list-based questions helped people better understand our questions.

By using a mix of open-ended and list-based questions, we were able to significantly improve data quality. MDD-W round 1In the first few rounds, we had an unusually high number of respondents who either scored “0” or “10” on the MDD-W score, which are both unlikely under normal circumstances. A score of “0” means that the respondent did not consume food items from any of the 10 core food groups the previous day or night, while a score of “10” means that the respondent consumed food items from all food groups. In the first round, scores of “0” or “10” accounted for 29 percent of all MDD-Wrespondents, but by Round 5 these scores represented only 3 percent of responses. It seems that having respondents reflect about what they ate in the open-ended questions we introduced in later rounds helps them  recall the food items they consumed and answer the subsequent list-based questions more accurately.

2. Keep questions simple.

We originally asked people by SMS whether they ate food items from the core food groups that comprise the MDD-W score. For example, “Yesterday, did you eat any Vitamin A-rich fruits and vegetables such as mangos, carrots, pumpkin, …….” Perhaps respondents thought that they needed to consume food items from both the fruit and vegetable groups in order to reply “yes” to this question. So instead, we split that question into two separate questions (one on Vitamin A-rich fruits and the other on Vitamin A-rich vegetables) to make it easier for the respondent to answer. We did the same for some of the other questions and found a very low percentage of women scoring “0” or “10” on the MDD-W score. Of course there is a trade-off here, and splitting too many questions might lead to a long and unwieldy questionnaire that could frustrate respondents.

3. Let respondents take the survey in their preferred language.

Comprehension remains a challenge in automated surveys, so helping respondents by asking questions in their own language will ensure data quality and limit non-response. In the Malawi study, translating food items into the local language (Chichewa), while keeping the rest of the questionnaire in English, improved comprehension. We recommend providing the respondent with the option to take the survey in their preferred language.

4. Pre-stratify and pre-target to ensure representativeness.

SMS surveys tend to be biased towards people who have mobile phones; we reach a lot of younger, urban men, and relatively few women of reproductive age, our target group for surveys on women’s diet. To ensure we are reaching them, an MDD-W SMS survey should be designed or ‘pre-stratified’ to include a diverse group of respondents. In Malawi, we were able to pre-stratify according to variables that included age, level of education, location and wealth. This allowed us to include women from all walks of life.

5. Post-calibrate to produce estimates that are more comparable to face-to-face surveys.

The MDD-W SMS surveys we conducted produced higher point estimates than those we would expect in face-to-face surveys. This suggests we may wish to consider calibration to adjust for sampling bias, the likely cause for the discrepancy. Calibration is the process of maintaining instrument accuracy by minimizing factors that cause inaccurate measurements. We’re still working on this and hope to find a solution soon. In the meantime, we think we are able to track trends in MDD-W by SMS with some reliability.

 

What we found at the market: using Free Basics in Malawi

FreeBasicsAd_Chichewa

We wrote to you back in November about one of our new innovations – our Free Basics website ‘Za Pamsika’ where we’re posting commodity prices using the weekly price data we’re collecting through our mVAM operators on a free website. We said that the project had the potential to reach millions of Malawians – well, a lot has happened since then.

Rather than continuing to willfully upload prices while watching our user statistics go up and down, we went to Malawi to carry out a short ground truth study and get some first hand user feedback.  The aim of the mission was to investigate the best way of using the website and interrogate the assumptions we’d made when designing it.

With this in mind, we tried to answer two big questions:

  1. Who can access our website – what are the potential barriers and how can we work around them?
  2. Do Malawians really want a website where they can find out maize and beans prices?

So we went to rural and urban markets in the Central and Southern regions to speak to the mVAM traders and the consumers in their markets about their mobile phone usage and market activity and to get their feedback on the website.

What kind of answers did we get?

First – access issues. While you don’t need a smartphone to access the website we knew that mobile penetration in Malawi is low. So we were most worried about the prevalence of internet-enabled phones and network coverage. From our study we found out that while we aren’t going to be able to reach everyone in Malawi via a website, we can still communicate with people. Network coverage was a problem in some areas. However, overall we found that most of the traders had internet enabled phones or wanted to buy one. We also found that Malawi’s MNOs have been recently trying to push out better network coverage. All good news for future reach of the website.

Actually the biggest barrier was language and literacy. While English is the national language of Malawi, most of the literate people we spoke to were much more comfortable reading and writing in Chichewa because that’s what they were taught in. While they were very enthusiastic about the website content when it was explained to them, they found the initial design (all in English and text heavy) confusing and difficult to use. Luckily this is an easy change to make so we did a quick redesign of the website and translated it into Chichewa:

malawiblog1

With our new design we headed back into the markets and got much better feedback. Rather than just saying that they liked the website content they could really interact with it and were making comments on the different maize and beans prices.

The second barrier we found was digital literacy. Many of the people we spoke to had internet-enabled phones but either didn’t know how to use them or didn’t even realise that they had the internet on them! Unlike the language change this is not a quick fix. This was particularly prominent amongst the women we spoke to, none of whom were comfortable with mobile internet. We’re therefore going to partner with civil society organisations promoting digital literacy. WFP has a network of partners and farmers on the ground who they reach out to with climate information so we’re going to try and use these focal points to communicate our prices with vulnerable populations and communities who have limited access to information.

IMG_1205

But do Malawians really want a ‘Za Pamsika’ website?

It turned out that maize and beans prices really are something that people want to see on the website. The recent drought was on everyone’s minds and they were really emphasising how much of a difference getting a good price could make. People were also already using their phones to get prices – by calling their friends or other traders in different areas and were quite enthusiastic about the possibility of getting this information for free.

With these learnings in mind and feeling confident with our website redesign and excited to be working closely with the country office, we embarked on our next steps. We now have a new focal point in the Lilongwe office who’s looking after the project and in a much better placed position than us in Rome to reach out to millions of Malawians. By this point over 25,000 people had already visited the Za Pamsika website but we knew our reach could be much further. We therefore started experimenting with ways of advertising the website.

malawiblog2

First – we decided to take out a Facebook ad to try and raise the site profile so we created our own ‘Za Pamsika’ page on Facebook and put out some ads in English and Chichewa. We were pretty excited when they started showing up on Malawian colleagues’ Facebook newsfeeds and within 10 days we’d reached more than 130,000 people and got 650 likes to our Facebook page.

What we didn’t expect was the organic reaction we’d get to our page. Within 3 days we’d not only reached more than 80,000 people with our post, we’d also seen that people started having conversations about maize prices on our advert.  People have also started messaging us about whether we can add their market to our website. We’re also getting comments about what other commodities we should add, for instance more seasonal foods such as groundnuts or soya. Most excitingly we even had someone knock on the door of the sub-office to inquire about the website after seeing our advert!

On a second mission in April we went out to the markets in Lilongwe again armed with our new ‘Za Pamsika’ posters. We were putting them up in the trader’s shops and were pretty quickly swamped with people excited about the website and how it could save them money. But again – everyone was asking us to add more food prices to the site – it seems like Malawians just keep wanting to know more about ‘things you find in the market’!

IMG_1201

So what’s next for Za Pamsika?

We’ve got our new focal point Khataza on board who’s taking charge of the website. First up, taking requests into account, we will be adding other seasonal commodities to the website. We’re going to continue experimenting with our Facebook ads and start using our Facebook page to reach out and engage with people about what they’d like on the page. We’ve also got some new partnerships coming up with civil society organisations who are keen to spread the word about ‘Za Pamsika’ and who we can work with to break down access barriers to this information.

Are millions in Malawi being reached? Not yet – but we’re getting there.

Can we reach rural women via mobile phone? Kenya case study

SONY DSC

WFP/Kusum Hachhethu

 

A few months ago, we published a blog post on our plans to test collecting nutrition data through SMS in Malawi and through live voice calls in Kenya. We just got back from Kenya where we conducted a large-scale mode experiment with ICRAF to compare nutrition data collected face-to-face with data collected through phone calls placed by operators at a call center. But before we started our experiment, we did a qualitative formative study to understand rural women’s phone access and use.

We traveled to 16 villages in Baringo and Kitui counties in Kenya, where we conducted focus group discussions and in-depth interviews with women. We also conducted key informant interviews with mobile phone vendors, local nutritionists, and local government leaders.

So in Kenya, can rural women be reached via mobile phone?

Here are the preliminary findings from our qualitative study:

  1. Ownership: Women’s phone ownership is high in both counties. However, ownership was higher in Kitui than Baringo, which is more pastoralist. From our focus group discussions and interviews, we estimate that 80-90% of women own phones in Kitui and 60-70% own phones in Baringo.
  1. Access: The majority of women had access to phones through inter- and intra-household sharing even if they didn’t own one themselves. This suggests that even women who don’t own a phone personally have access to phones that they may be able to use to participate in phone surveys.
  1. Usage: Women mostly use phones to make and receive calls, not send SMS. This supports our hypothesis that voice calls, not SMS, would be the optimal modality to reach women through mobile surveys.
  1. Willingness: Women were enthusiastic about participating in phone surveys during our focus group discussions and in-depth interviews, implying that they are interested in phone surveys and willing to take part.
  1. Trust: Unknown numbers create trust issues, but they are not insurmountable. Women voiced concerns about receiving phone calls from unknown numbers. Despite these trust issues, we were eventually able to successfully conduct our phone surveys after sensitizing the community, using existing community and government administration structures.
  1. Network: Poor network coverage, not gender norms or access, is the biggest barrier to phone surveys in the two counties. Women identified network coverage as the biggest barrier for communication. Some parts of the counties had poor to no network coverage. However, we found that phone ownership was high even in these areas, and women would travel to find network hotspots to make or receive phone calls.

So in conclusion, yes, in Kenya it is possible to reach rural women by phone.
Our findings from Kitui and Baringo counties show that we can reach women in similar contexts with mobile methodologies to collect information on their diet as well as their child’s diet.

We are also analysing the quantitative data from our mode experiment to find out whether data on women and children’s diet collected via live phone operators gives the same results as data collected via traditional face-to-face interviews.

The El Niño Aftermath: Tracking Hunger in the Millions in Southern Africa

We’ve been writing a lot about how mVAM can help in conflict situations where whole areas are cut off because of violence or an epidemic (see our blogs on Yemen, Somalia, Iraq and article on Ebola). But over the past year, the world was disrupted by another type of event- a climatic one: El Niño. The El Niño weather pattern results from a warming of sea temperatures in the Pacific roughly every three to seven years. This El Niño was one of the strongest on record.  The reason why El Niño was so concerning is its global reach, it didn’t just affect the Pacific; places as far away as Guatemala, Pakistan, Indonesia and Ethiopia were all at risk of floods and/or droughts. While the El Niño itself has abated, it has left millions hungry in its wake (current estimates are that 60 million people are food insecure globally). And a La Niña year is looming.

One area that has been particularly affected is Southern Africa. Across the region, this year’s rainfall season was the driest in the last 35 years. Most farmers are facing significantly reduced and delayed harvests.

El Niño hit when Southern Africa was already vulnerable to food insecurity. The region had already experienced a poor 2014-15 harvest season, meaning that food stocks were already depleted. Now, after El Niño, roughly 41 million people are classified as food insecure. On 13 June 2016 WFP categorized the region as an L3 emergency – a situation requiring the highest level of humanitarian support. We’re therefore dramatically expanding our national food security monitoring in the region so WFP can quickly provide as much relevant food security information as possible to effectively respond to the crisis.

Predictions that this El Niño would have a big effect had already started coming in 2015 so we began setting up mobile monitoring in countries that were particularly vulnerable to El Niño. We started in Malawi which had very disruptive weather patterns looming (potentially too much rain in the north and huge rainfall deficits in the south). We lacked current household data to track the impact on food security across the country.

To get information quickly and cheaply, we started a monthly SMS survey with GeoPoll in December 2015. And Malawians sure were quick to respond! In 24 hours, we had 1,000 questionnaires completed.  When analyzing the results, we wanted to make sure people were understanding our texts. The adult literacy rate in Malawi is only 61.3% so we kept the questionnaire short and as simple as possible. We included questions for one food security indicator- the reduced coping strategy index (rCSI) which asks people about the coping strategies they are using when they don’t have enough to eat. We also checked that the data made sense, and in general, the rCSI behaved as we would suspect. It was correlated with people’s messages about their community’s food security situation and their wealth status. As with all of our surveys, we are continually improving them. In this case, we increased our sample size and district quotas to capture more people in rural areas.

Monitoring Maize Prices

IMG_0095Market prices, especially maize prices, are key to Malawians’ food security. Maize is the staple food, used to make nsima which is consumed daily. So to monitor market prices in 17 hotspot districts, we collected phone numbers from over 100 traders in 51 markets throughout Malawi. We first tried asking them prices by text message, but we didn’t receive many responses.  It seems like sending back a series of texts is a bit too much to ask of traders who volunteered out of their own good will to participate in our market survey. We therefore set up a small call center in WFP’s country office. We trained two operators, and they were quickly placing calls to traders every week. When they could just answer a quick phone call instead of having to type in answers, traders willingly reported current commodity prices.

Our latest report from June 2016 shows that maize prices are now between 50 and 100 percent higher than this time last year. This is having a big effect on Malawians. As you can see from our word cloud, alarmingly ‘not-enough’ featured prominently in our open ended question about maize.

word cloud_cropped

Nutrition Surveillance for the first time

In most countries, we have been concentrating on household level indicators like food consumption. But health centers treating malnutrition could potentially give us important indications of the nutrition situation of different parts of the country. In Malawi, WFP works with health centers to address moderate acute malnutrition (MAM) in Malawi by providing fortified blended foods. So to make the most of our call center, we decided to call these health centers every two weeks and track malnutrition admission data for children (aged 6-59 months) and for adults with HIV/AIDS or tuberculosis. In the first six weeks of monitoring, we saw a big increase in the number of moderate acute malnutrition admissions for children increased greatly where severe acute malnutrition rates did not show a clear pattern. We dug further, and the Ministry of Health had initiated mass screenings to enroll malnourished children in nutrition programmes which generally pick up moderately malnourished children. With health center admission data, it’s important to check what else is going on in the country. We’re hoping to soon pilot contacting mothers of malnourished children about their children’s progress to gain additional insight into the nutrition status of vulnerable populations in Malawi.

Now that we have Malawi firmly established, we’ve started reporting on Madagascar and our data collection is ongoing in Zambia, Lesotho and Mozambique. So watch this space for more news about how we get on in these next few months.