Postcard from Dakar

mVAM workshop participants all smiles after learning more about IVR WFP/Lucia Casarn

mVAM workshop participants all smiles after learning more about IVR WFP/Lucia Casarn

During the last week of June, staff from WFP HQ’s mVAM team, the West and Central Africa Regional Bureau, and Nigeria and Niger Country Offices met in beautiful Dakar to work together on Interactive Voice Response (IVR) systems for two-way communication. (If you want to dig deep into all details IVR-related, check out the lesson in our mVAM online course!)

We’ve previously blogged about how WFP is responding to the needs of people who have been displaced due to Boko Haram insurgencies in both Nigeria and Niger. When we implemented these operations we also put communication channels in place so beneficiaries are able to contact WFP. In Nigeria, the Maiduguri Field Office created a hotline. Their operators receive an average of 100 calls per day from beneficiaries asking food security-related questions and providing feedback on the operations. The problem is the hotline is only available during working hours and has a limited number of people who can call in at the same time. To work around this they’re therefore looking at how an IVR system can support the call operators who are dealing with high volumes and better manage calls that take place outside of normal office hours. WFP Niger wants to set up a similar hotline system but without full time phone operators. Beneficiaries will call in to an automated IVR system and their queries and feedback recorded by the system and followed up by the Country Office. 

A Nigeria IT Officer working to install a GSM gateway for IVR usage in Maiduguri WFP/Lucia Casarin

A Nigeria IT Officer working to install a GSM gateway for IVR usage in Maiduguri WFP/Lucia Casarin

During the workshop participants were trained by InSTEDD on how to physically deploy IVR using a GSM gateway (a fancy tool that automatically places phone calls) and Verboice, the free open source software they’ve developed to manage these systems. The team also discussed the nitty gritty technical aspects of the system, including creating and modifying call flows (the sequencing of questions), scheduling calls and downloading collected call logs and recordings. Most importantly, participants had the opportunity to share their experiences and challenges with experts in this field and discuss best practices, alternative deployments and technical solutions.

The Country Office staff have now returned to Niger and Nigeria and they’ve already started testing the use of the IVR machines. We’re eager to begin logging data and hearing more from our beneficiaries. So stay tuned!

 

 

 

Trial and Error: How we found a way to monitor nutrition through SMS in Malawi

WFP/Alice Clough

WFP/Alice Clough

Over the last ten months we have been testing if we can use mobile phones to collect nutrition indicators. One of these experiments involved using SMS to ask questions about women’s diet quality via the Minimum Dietary Diversity – Women (MDD-W) indicator.  The MDD-W involves asking simple questions about whether women of reproductive age (15-49 years) consumed at least five out of ten defined food groups. We were interested in using SMS surveys to measure MDD-W, because SMS offers an opportunity to collect data regularly at scale and at low cost.

From October 2016 to April 2017, we worked with GeoPoll to conduct five survey rounds on MDD-W and find a way to adapt the indicator to SMS. We analysed data from each round, identified gaps and refined the survey instrument. We were able to collect data quickly and identify strengths and weaknesses to make revisions through an iterative process. Through this process, we believe that we have successfully designed an instrument that can be used to monitor MDD-W trends by SMS. Here’s a short summary of what we learned:

1. Using a mix of open-ended and list-based questions helped people better understand our questions.

By using a mix of open-ended and list-based questions, we were able to significantly improve data quality. MDD-W round 1In the first few rounds, we had an unusually high number of respondents who either scored “0” or “10” on the MDD-W score, which are both unlikely under normal circumstances. A score of “0” means that the respondent did not consume food items from any of the 10 core food groups the previous day or night, while a score of “10” means that the respondent consumed food items from all food groups. In the first round, scores of “0” or “10” accounted for 29 percent of all MDD-Wrespondents, but by Round 5 these scores represented only 3 percent of responses. It seems that having respondents reflect about what they ate in the open-ended questions we introduced in later rounds helps them  recall the food items they consumed and answer the subsequent list-based questions more accurately.

2. Keep questions simple.

We originally asked people by SMS whether they ate food items from the core food groups that comprise the MDD-W score. For example, “Yesterday, did you eat any Vitamin A-rich fruits and vegetables such as mangos, carrots, pumpkin, …….” Perhaps respondents thought that they needed to consume food items from both the fruit and vegetable groups in order to reply “yes” to this question. So instead, we split that question into two separate questions (one on Vitamin A-rich fruits and the other on Vitamin A-rich vegetables) to make it easier for the respondent to answer. We did the same for some of the other questions and found a very low percentage of women scoring “0” or “10” on the MDD-W score. Of course there is a trade-off here, and splitting too many questions might lead to a long and unwieldy questionnaire that could frustrate respondents.

3. Let respondents take the survey in their preferred language.

Comprehension remains a challenge in automated surveys, so helping respondents by asking questions in their own language will ensure data quality and limit non-response. In the Malawi study, translating food items into the local language (Chichewa), while keeping the rest of the questionnaire in English, improved comprehension. We recommend providing the respondent with the option to take the survey in their preferred language.

4. Pre-stratify and pre-target to ensure representativeness.

SMS surveys tend to be biased towards people who have mobile phones; we reach a lot of younger, urban men, and relatively few women of reproductive age, our target group for surveys on women’s diet. To ensure we are reaching them, an MDD-W SMS survey should be designed or ‘pre-stratified’ to include a diverse group of respondents. In Malawi, we were able to pre-stratify according to variables that included age, level of education, location and wealth. This allowed us to include women from all walks of life.

5. Post-calibrate to produce estimates that are more comparable to face-to-face surveys.

The MDD-W SMS surveys we conducted produced higher point estimates than those we would expect in face-to-face surveys. This suggests we may wish to consider calibration to adjust for sampling bias, the likely cause for the discrepancy. Calibration is the process of maintaining instrument accuracy by minimizing factors that cause inaccurate measurements. We’re still working on this and hope to find a solution soon. In the meantime, we think we are able to track trends in MDD-W by SMS with some reliability.

 

Postcard from Bangui

Good to be OKING:It may not be new and super large, but the owner claims this phone has a week-long battery life! WFP/Dominique Ferretti

It may not be new and super large, but the owner claims this phone has a week-long battery life!
WFP/Dominique Ferretti

Greetings from the Central African Republic (CAR)! Our team recently visited Bangui and Kaga-Bandoro to help the Country Office team assess how to enhance the current mVAM system and see what other mVAM technologies we might be able to deploy. CAR is a very unique context, because there’s little-to-no cell phone reception outside of main towns. Only 26% of the population own a phone, one of the lowest rates in the world according to the World Bank.  This means that collecting data remotely takes some creativity. The CAR team uses a key informant system, where they contact approximately 200 people around the country each month to collect information on basic commodity prices, market access, population movements, and security issues. The collected information is then shared with the humanitarian community, who appreciate the data, as it’s the only national-level food security data that’s currently collected regularly!

A local woman in Kaga-Bandoro selling a great source of protein and a central African delicacy—caterpillars! WFP/Dominique Ferretti

A local woman in Kaga-Bandoro selling a great source of protein and a central African delicacy—caterpillars!
WFP/Dominique Ferretti

The only downfall to the key informant system is that it doesn’t give us household-level food security information. The CAR team has therefore decided to try a small pilot using household questionnaires in the city of Kaga-Bandoro. Courtesy of UNHAS, we visited the city (more like a very small town!) and the 2 IDP camps it hosts during our day trip. While not that many people had cell phones, enough community members and displaced persons had phones that we’ll be able to get some idea of the food security situation.

Stay tuned for more as the pilot unfolds…!

Chatting with community members as they collect water WFP/Dominique Ferretti

Chatting with community members as they collect water
WFP/Dominique Ferretti

If you’re not human then who are you?

Experimenting with chatbots in Nigeria and Haiti

WFP/Lucia Casarin

Testing the bot in Haiti – WFP/Lucia Casarin

Readers of this blog know that the team has been experimenting with chatbots to communicate with disaster-affected communities – read our previous posts about our prototype and the Nielsen Hackathon.

As part of this effort, during recent missions to Haiti and Nigeria, our team went out to talk to communities to find out whether a chatbot would be right for them.

Would a chatbot be a stretch in these communities?

Well it’s not that much of a stretch.

In North East Nigeria, most displaced people live in Maiduguri, a city of over 1 million people. In this ‘urban’ setting connectivity is good, most people own cell phones and many young people use social media and messaging apps. Mobile operators have been offering services that allow people to access the internet by selling ‘social bundles’ (unlimited social media access sold in very small increments) and offer some services for free, including Facebook Light and Facebook Messenger.

In Haiti, three-quarters of the population live in the capital, Port-au-Prince, where 3G connectivity is good and most people use messaging apps to communicate with friends and family. Even in rural and difficult-to-reach communities, leaders and young people own smartphones and connect to the internet. There is a lot of competition between mobile operators so the prices for mobile data are very low. This means that most people can afford to access the internet either via their own smartphone or from shared smartphones.

A

Mobile phones charging station on the road from Léogane Peri to Port-au-Prince WFP/Lucia Casarin

A bare-bones demo

In both countries we tested a simple chatbot that asks people about food prices and what the food security is like in their community. The survey we used was much more basic than our usual mobile questionnaires as we felt it was important to keep things simple at this stage.

For Nigeria, the bot demo was initially in English but we soon translated it into Hausa, the primary language spoken by displaced persons in Maiduguri. In Haiti we made it available both in Creole and French. The chatbot was very responsive on 3G and it even worked with slower 2G connections so the technology works in these contexts. But this was only the starting point, what we really wanted to know was what ‘real’ people thought about the bot.

We organized focus group discussions with displaced people in Maiduguri and with community representatives in Haiti. We helped people access the WFP bot via their Facebook accounts, and they began chatting away.

Sounds cool, but what are the limitations?

Here’s what people said:

First of all, people thought the bot is a convenient, quick, and easy way to get in touch directly with WFP and they really liked that the bot allows them to speak to WFP without intermediaries. They had lot to tell us particularly through the open-ended question where they typed out detailed responses.

In Nigeria, they did tell us that our (somewhat wordy) English-language demo should be translated into Hausa because it would make it easier for everyone to use. Our first group of testers were young people who were already Facebook users and so were familiar with Messenger. It was therefore no surprise that they were interacting smoothly with the bot and able to go through our questionnaire in minutes.

WFP/Jean-Martin Bauer

Testing the bot in Nigeria – WFP/Jean-Martin Bauer

In Haiti, people started interacting with the bot as if it was a human rather than an automated questionnaire so they got stuck pretty fast when it wasn’t as naturally responsive as they’d expected. This means that either we give clearer instructions to people or we add Natural Language Processing capabilities to our bot.

There are of course other barriers. In both countries women appeared to be less likely to own a smartphone. This means that bot users will likely be overwhelmingly young, male and better educated than other people – hardly ‘representative’ of WFP’s target affected population. The free version of the bot is also not always available: in Nigeria only Airtel subscribers can access it, while in Haiti the free service doesn’t exist yet.

This means that the bot would need to be a complement to the other tools we have. We might use data from the bot to obtain a quick situation update, but we will continue relying on other sources for more representative data.

mVAM for nutrition: findings from Kenya

2WFP-Kusum_Hachhethu

Photo: WFP/Kusum Hachhethu

We’ve used mVAM to collect data on a range of things that impact food security – so what about information on nutrition? Back in October, we went to Kenya to conduct a study on whether we could use remote mobile data collection to gather information on women and children’s nutrition.

The summary of our findings from the case study are now available in a new report from mVAM and our partners in the study, WFP’s Nutrition Division and the World Agroforestry Centre (ICRAF).

Read more:

kenya-report

What we found at the market: using Free Basics in Malawi

FreeBasicsAd_Chichewa

We wrote to you back in November about one of our new innovations – our Free Basics website ‘Za Pamsika’ where we’re posting commodity prices using the weekly price data we’re collecting through our mVAM operators on a free website. We said that the project had the potential to reach millions of Malawians – well, a lot has happened since then.

Rather than continuing to willfully upload prices while watching our user statistics go up and down, we went to Malawi to carry out a short ground truth study and get some first hand user feedback.  The aim of the mission was to investigate the best way of using the website and interrogate the assumptions we’d made when designing it.

With this in mind, we tried to answer two big questions:

  1. Who can access our website – what are the potential barriers and how can we work around them?
  2. Do Malawians really want a website where they can find out maize and beans prices?

So we went to rural and urban markets in the Central and Southern regions to speak to the mVAM traders and the consumers in their markets about their mobile phone usage and market activity and to get their feedback on the website.

What kind of answers did we get?

First – access issues. While you don’t need a smartphone to access the website we knew that mobile penetration in Malawi is low. So we were most worried about the prevalence of internet-enabled phones and network coverage. From our study we found out that while we aren’t going to be able to reach everyone in Malawi via a website, we can still communicate with people. Network coverage was a problem in some areas. However, overall we found that most of the traders had internet enabled phones or wanted to buy one. We also found that Malawi’s MNOs have been recently trying to push out better network coverage. All good news for future reach of the website.

Actually the biggest barrier was language and literacy. While English is the national language of Malawi, most of the literate people we spoke to were much more comfortable reading and writing in Chichewa because that’s what they were taught in. While they were very enthusiastic about the website content when it was explained to them, they found the initial design (all in English and text heavy) confusing and difficult to use. Luckily this is an easy change to make so we did a quick redesign of the website and translated it into Chichewa:

malawiblog1

With our new design we headed back into the markets and got much better feedback. Rather than just saying that they liked the website content they could really interact with it and were making comments on the different maize and beans prices.

The second barrier we found was digital literacy. Many of the people we spoke to had internet-enabled phones but either didn’t know how to use them or didn’t even realise that they had the internet on them! Unlike the language change this is not a quick fix. This was particularly prominent amongst the women we spoke to, none of whom were comfortable with mobile internet. We’re therefore going to partner with civil society organisations promoting digital literacy. WFP has a network of partners and farmers on the ground who they reach out to with climate information so we’re going to try and use these focal points to communicate our prices with vulnerable populations and communities who have limited access to information.

IMG_1205

But do Malawians really want a ‘Za Pamsika’ website?

It turned out that maize and beans prices really are something that people want to see on the website. The recent drought was on everyone’s minds and they were really emphasising how much of a difference getting a good price could make. People were also already using their phones to get prices – by calling their friends or other traders in different areas and were quite enthusiastic about the possibility of getting this information for free.

With these learnings in mind and feeling confident with our website redesign and excited to be working closely with the country office, we embarked on our next steps. We now have a new focal point in the Lilongwe office who’s looking after the project and in a much better placed position than us in Rome to reach out to millions of Malawians. By this point over 25,000 people had already visited the Za Pamsika website but we knew our reach could be much further. We therefore started experimenting with ways of advertising the website.

malawiblog2

First – we decided to take out a Facebook ad to try and raise the site profile so we created our own ‘Za Pamsika’ page on Facebook and put out some ads in English and Chichewa. We were pretty excited when they started showing up on Malawian colleagues’ Facebook newsfeeds and within 10 days we’d reached more than 130,000 people and got 650 likes to our Facebook page.

What we didn’t expect was the organic reaction we’d get to our page. Within 3 days we’d not only reached more than 80,000 people with our post, we’d also seen that people started having conversations about maize prices on our advert.  People have also started messaging us about whether we can add their market to our website. We’re also getting comments about what other commodities we should add, for instance more seasonal foods such as groundnuts or soya. Most excitingly we even had someone knock on the door of the sub-office to inquire about the website after seeing our advert!

On a second mission in April we went out to the markets in Lilongwe again armed with our new ‘Za Pamsika’ posters. We were putting them up in the trader’s shops and were pretty quickly swamped with people excited about the website and how it could save them money. But again – everyone was asking us to add more food prices to the site – it seems like Malawians just keep wanting to know more about ‘things you find in the market’!

IMG_1201

So what’s next for Za Pamsika?

We’ve got our new focal point Khataza on board who’s taking charge of the website. First up, taking requests into account, we will be adding other seasonal commodities to the website. We’re going to continue experimenting with our Facebook ads and start using our Facebook page to reach out and engage with people about what they’d like on the page. We’ve also got some new partnerships coming up with civil society organisations who are keen to spread the word about ‘Za Pamsika’ and who we can work with to break down access barriers to this information.

Are millions in Malawi being reached? Not yet – but we’re getting there.

Hearing from those who are #FacingFamine

Photo: WFP/Amadou Baraze

Photo: WFP/Amadou Baraze

 

In early March, Stephen O’Brien, the United Nations’ Emergency Relief Coordinator, reported that 20 million people across four countries face starvation and famine.  The famines looming in Yemen, South Sudan, Somalia and Nigeria represent the largest humanitarian crisis since the UN’s creation. “Without collective and coordinated global efforts,” O’Brien said, “People will simply starve to death, and many more will suffer and die from disease.”

One of the components that complicates these particular emergencies is access to the areas in crisis. Without safe and unimpeded access for humanitarian aid workers, it’s difficult to get a picture of what’s going on in the affected areas, which adds another dimension to an already challenging response. In Northeast Nigeria, the threat of violence made it difficult for WFP’s food security analysts to visit vendors in local markets or speak with people in their homes – all part of their usual food security monitoring routine.

In order to continue gathering information needed to understand the situation in the affected areas, WFP used remote mobile data collection to get a picture of what was happening in the communities they could no longer speak to in person. With an overwhelming amount of responses, we turned to Tableau , who had already helped us create data visualizations for other countries which use mVAM, to help us visualize the results in a way that could be easily understood by everyone.

mVAM hears directly from people in affected communities in the northeast of Nigeria

mVAM hears directly from people in affected communities in the northeast of Nigeria

 

Our latest interactive data visualization of the food security situation in Northeast Nigeria is now online, and the story of how it came to be can be found on Tableau’s blog. Make sure to check out the free response section, where you can hear from 5,500 households on what should be done to improve the food security in their community.

 

Myanmar: assessing emergency needs without access

Photo: WFP/Myanmar

Photo: WFP/Myanmar

 

Late last year, an attack by an armed group on border police posts in Myanmar led to a government security sweep in Rakhine State and recurrent clashes and violence in many villages. As a result, access to a large part of the north of the state was closed off to humanitarian organizations, leaving the already highly vulnerable inhabitants of the townships to fend for themselves.

Unable to access the area since 9 October, WFP decided to use mobile surveys to conduct remote emergency assessments. While not as thorough as face-to-face assessments, mobile surveys could still provide a good snapshot of how people were coping in the areas that were closed off. Furthermore, mobile surveys serve as a means to address a critical information gap where there is little to no information about the needs of the most vulnerable and food insecure, as we have seen in complex emergency settings elsewhere such as during the Ebola crisis and Yemen. But let’s come back to Myanmar and rewind just a few years: hearing from people in these areas would have been impossible – essentially no one had mobile phones.

Myanmar’s mobile transformation

Myanmar’s telecommunication market has come a long way. Not so long ago, Myanmar was one of the “leastconnected countries in the world” – just seven years ago, SIM cards cost up to $1,500, and few people had them. In 2013, after the government awarded contracts to two foreign mobile operators, the price of a SIM card fell to $1.50 and network coverage began to roll out across the country. Once the mobile revolution began, things moved fast. Soon, mobile penetration exceeded even that of much better-off neighboring countries, such as Thailand[1]. By 2015, 96 percent of wards and 87 percent of villages in Myanmar had a mobile signal, and nearly 60 percent of households owned a mobile phone[2].

A case for mobile surveys in Myanmar

WFPs first mobile assessment in Myanmar took place in November 2016, with 32 key informants from 12 villages in Maungdaw and Buthidaung north, complementing face-to-face interviews of 48 WFP beneficiaries at 8 food distribution points in Buthidaung south. This was at the end of the lean season (the period between harvests when households’ food stocks tend to be the lowest), and respondents told us that due to the deteriorated security situation, people faced serious difficulties in reaching markets, were not able to go to work, nor access agricultural land and fishing areas and. Resulting crop losses could result in mid to long-term impact on food security while households’ terms of trade had decreased and posed a serious concern regarding their ability to purchase sufficient food.

Though low mobile penetration in rural areas of the country posed a challenge for phone surveys, people were nonetheless eager to participate in the survey and share their stories. In order to participate, some people even arranged to borrow phones from neighbors if they did not own one themselves.

A second phone survey in December allowed for a greater sample size and therefore a better understanding of the living conditions in the surveyed areas. WFP spoke to 116 respondents in 70 villages in Maungdaw Township. By this time, the people we spoke with mentioned that there was widespread food insecurity throughout the township. The situation was particularly problematic in the north, where markets were not functioning and access to agricultural land or fishing grounds was restricted. Livelihood opportunities were scarce and the lower demand for daily labour had had an immediate impact on the most vulnerable.

Photo: WFP/Myanmar

Photo: WFP/Myanmar

What’s next?

The data collected through the phone surveys helped WFP to get some understanding of the needs in the no-access areas, and to use this information for advocacy with the Government and humanitarian stakeholders. On 9 January 2017, after three months, WFP was finally granted access to all areas where it had operations prior to 9 October, and was able to distribute food to 35,000 people in the villages of Maungdaw north. With the area open again, WFP and its partners are now preparing for thorough assessments on the ground, which will give a fuller picture of the food security situation and also allow us to validate the findings of the phone surveys.


[1]http://lirneasia.net/wp-content/uploads/2015/07/LIRNEasia_MyanmarBaselineSurvey_DescriptiveStats_V1.pdf

[2]http://www.gsma.com/mobilefordevelopment/wp-content/uploads/2016/02/Mobile-phones-internet-and-gender-in-Myanmar.pdf

Mind the Mode 2: Settling the (Food Consumption) Score in South Sudan

POC3_Nektarios_Markogiannis

POC 3
Photo: UNMISS/Nektarios Markogiannis

For the second installment of our ‘mind the mode’ series, we’re taking you to Juba, South Sudan, where we previously conducted a mode experiment. What we wanted to see was how food security indicators compare when data is collected face-to-face and through operators over the phone.

South Sudan is a complex setting for mobile surveys to begin with. The country has low cell phone penetration- it’s estimated to be only 20%. Network quality is a problem, often calls don’t go through or audio is poor.  Last, but not least, the country has been extremely unstable. While we have been using key informant phone interviews to date, we are investigating the feasibility of conducting phone surveys to collect household food security indicators. Given the complexities, starting with a test to evaluate biases related to survey mode seemed prudent.

Methodology

The mode experiment took place in “POC 3”, a Protection of Civilians (POC) camp in Juba near the main UN compound. POC 3 is the largest of three camps at the UN House site in Juba, with an estimated population of 20,000 people, according to the International Organization for Migration. People in the POC are there in search of protection against the violence and conflict that South Sudan has been experiencing. We’re hoping to use mobile phones to monitor food security indicators in POC communities. POC 3 happens to have good cell phone coverage – a 2014 survey estimated that some 70% of households in the camp had access to a phone.  

 

Photo: WFP/Silvia Passeri

Photo: WFP/Silvia Passeri

We evaluated how mode effects the Food Consumption Score (FCS), which measures the frequency of consumption of different food groups consumed by a household during the 7 days before the survey. A higher score means a better level of the respondent’s household food security. The FCS is a commonly used proxy for household food security.

We carried out two rounds of data collection, round 1 in March and round 2 in May 2016. In round 1, half of the respondents received a voice call survey and the other half participated in an identical interview face-to-face. The ‘treatment’ (voice call) was random. In round 2, some of the respondents that received a voice call took the exact same survey face-to-face, and vice versa.

There were challenges relating to security in the POC and some of the respondents from March were not found in the camp when we conducted the second round in May. As a result, we had 132 voice and 333 face-to-face interviews in round one, but 138 voice and only 117 face-to-face surveys in round 2. This sample size is smaller than we would have liked, but we think it’s indicative enough to tell us how responding to a phone survey differs from one that took place face-to-face.

Calls were placed by operators that were ‘converted’ enumerators – field monitors who usually carry out WFP’s post-distribution monitoring but were new to phone-based surveys. This meant that they were already familiar with the food security indicators and the camp community, but needed training on the protocol for phone-based surveys.

Results

We observed substantial mode effects in round 1. We obtained a mean FCS of 34 via face-to-face surveys, but a much higher score of 45  through voice calls. Our regression analysis shows that mode alone accounted for 7 points in the difference in a household’s response (p<0.01), with other factors accounting for the remainder of the difference. This means that a voice survey would inflate the FCS by 20%, leading to a gross underestimation of the severity of food insecurity in the population of interest. During round 1, the voice FCS question behaved as an almost binary variable – we would get 1s and 7s, but very few 2,3,4,5 answers. That means a lot of people said they ate a given food item one day or every day, but that very few other answers were being recorded.

FCS results, round 1

FCS results, round 1

In round 2, the difference between voice calls and face to face surveys diminished substantially. Also, the difference was not statistically significant. In fact, the slight remaining difference between the two groups was due to respondent households’ socio economic profile, not because of the mode we used to collect data.

 

R2

FCS results, round 2

Lessons learned

For the food consumption score, the differences between voice and face-to-face due to the mode effect were large in round 1, but vanished in round 2. This is a positive finding for us as we are seeking to rigorously test and validate the data collected through mobile and reporting on the results with some degree of confidence. We want to highlight a few lessons here that could help guide others into the right direction.

Lesson 1: Practice makes perfect.  We suspect that the poor quality of the data collected in round 1 is due to our call center being brand new, and experiencing ‘teething’ problems. When an in-house call center is first set up, it tends to be small scale comprising of one or two operators. With resources permitting (and provided there is increased information needs) the call center may be expanded with additional operators who will receive regular training and coaching. Our analysts have been saying anecdotally that data quality improves as time goes by and the system becomes more established. We have a good illustration of the phenomenon here in South Sudan.

Lesson 2: Close supervision is required! Although our operators were familiar with data collection, it took time to train them to implement surveys by phone with quality.  This again shows that operator selection, training, and supervision are key to obtaining good quality data.

Lesson 3: Work with professional call centers. Overall, this encourages us to continue working with professional call centers when possible, and avoid the temptation to do things in-house in a hurry – something that can be all too tempting in an emergency setting.

We also think the method used in South Sudan could be applied elsewhere to help evaluate mode effects. We will post the survey design on the mVAM Resource Center for others to use.

How many pizzas does it take to build a chatbot?

Hackers are hungry Photo: WFP/Pia Facultad

Hackers are hungry
Photo: WFP/Pia Facultad

This week, government, business, academia and civil society leaders will gather at Davos to discuss solutions to the world’s biggest challenges – including how new technologies can be leveraged to solve some of the most serious problems we face. At mVAM, we continue to explore how some of these technologies could be used to help eliminate chronic hunger, malnutrition and food insecurity – most recently looking at how chatbots could help collect important information during a humanitarian response.

Last week, our collaborators at Nielsen – one of the early supporters of mVAM – organized a 24-hour hackathon at the Nielsen Tech Hub in New York City. As part of ongoing efforts through Nielsen Cares, the hackathon aimed to develop an open-source humanitarian chatbot that can collect real-time information about food security. This came at the right time for WFP – we’d developed and tested a prototype of the chatbot with InSTEDD, and Nielsen’s technology and development input helped bring in important new capabilities. Ultimately, our goal is to field-test a chatbot in Haiti in the next few months to help us track food security conditions as people recover from the impacts of Hurricane Matthew.

The event was open to the public. A diverse group of students, volunteer hackers, and Nielsen staff showed up to take on the challenge, despite the wintry weather. InSTEDD’s Director of Platform Engineering, Nicolás di Tada also participated.

Much more than a chatbot

What the hackers built is much more that a chatbot: it is a bona-fide chat-based data collection and reporting system. Rather than attempt to outdo each other (as is the case in most hackathons), the teams split up to build the different components of the system. The different teams, made up of perfect strangers, communicated during the hackathon through Slack. After 24 hours, most components were fully coded up, but there were still bugs with the orchestrator and the gateway that additional post-hackathon work will resolve.

The architecture of the system, as defined by Nielsen, includes:

  • a management interface that allows an analyst to set up a questionnaire, including and skip logic, and validation rules that prompt the user when they enter a wrong answer. The interface was built using the Angular 2 JavaScript framework;
  • a gateway that is able to interact with respondents through Facebook Messenger and potentially other chat applications. The Facebook gateway was built on top of the AWS Lambda service;
  • a natural language processing engine that analyzes text on the fly. It allows the chatbot to ‘interpret’ a user’s answers. For now, the NLP engine processes English language text, although the engine includes a translation service and, by default, translates all languages to English for more advanced NLP tasks. The engine was built using the AWS Lambda service and leverages IBM Watson’s AlchemyLanguage service for text processing.;
  • a set of ‘backend APIs’ that manage respondent and survey data, route respondents from each response to the next question, and provide data to user interfaces .  The APIs were built using the Django framework for python and deploys on the AWS Elastic Beanstalk service;
  • an ‘orchestration layer’ that maintains survey status and routes messages between the end user and the various backend services. The orchestration service is built on top of the AWS Lambda service; and
  • a “reporting and data visualization engine”. Data vizzes were built using Highcharts, a JavaScript-based application. This allows an analyst to instantly see the results of the chatbot surveys.

 

chatbot

 

Leveraging cloud services from the Amazon Web Services product catalog, the teams were able to build a scalable, cost effective platform that can be deployed quickly to multiple locations globally.

Remember the humans

We also received tips from a chatbot specialist, Alec Lazarescu from Chatbots Magazine. He encouraged us to ‘onboard’ users with an initial message that gives people a clear idea of what the chatbot is for. He told us to avoid ‘dead ends’ and allow users to speak to a human being in case they get stuck.

We’re very grateful to Nielsen for their support and to all the participants for their energy and creativity. The next steps involve WFP and InSTEDD accessing the code and work on ironing out the kinks. We expect challenges with the natural language processing in Haitian Creole, a language that is probably under-researched. Making the different parts of the chatbot work together seamlessly also appears to be an area we will still have to work on.  And, of course, the final test will be to see whether our target group – people living in Haiti – find the chatbot engaging.