Chatbot: back to the drawing board

alice-hand-model

We’ve recently developed a prototype of a chatbot to communicate with people via the Telegram messaging app, but it will eventually work on any messaging app. The purpose of a prototype is to test our approach thus far in the real world and then go back to the drawing board to improve it. Before this month, our testing had been limited to our colleagues here in Rome and our partners at InSTEDD.  However, we really needed feedback from people in the communities we are actually trying to reach.

Eventually we’ll test a later version of the chatbot in the countries where we work. But for some initial feedback, we were able to get in contact with people right on our doorstep, who had completed a difficult journey to Rome. UNHCR recently estimated that there are roughly 65.3 million people currently forcibly displaced worldwide. Instability and conflict in the Middle East and Africa has led many to flee to Europe in the hope of a safer, better life. One of the most used and most dangerous routes is via Libya and then across the Mediterranean Sea to southern Italy. To give you an idea of the scale, between 29 August and 4 September this year, Italy averaged over 2000 arrivals every day. Of those who reach the mainland, many make their way north to Rome. There are now many centres across the city that provide refuge, often in the form of meals, language learning and legal support.  

UN photo/UNHCR/Phil Behan

We went to one of these migrant centres to speak with people and get their feedback on whether the chatbot would be useful in their home communities. We can hear migration statistics but listening to people’s stories really made these statistics come alive. One person we talked to described being saved by the Italian Coast Guard as the ship transporting him sunk in the Mediterranean. He said he will always be grateful to the Italian government for his rescue.

So needless to say, we were very grateful that people would take time and test the bot. Its is currently in English so we were only able to test it with English speakers right now. The goal is to get it running well in English and then translate and adapt it to other languages.

First we asked people a couple of questions about smartphone ownership in their country of origin. They told us that while the poorest people in rural areas don’t have smartphones around 70% of the population does – meaning that we can still communicate with a lot of people via smartphone. They then had a go at using our chatbot, first answering the food security survey and then trying out the price database. Here are a few things that speaking with them helped us realize:

Simplify our questions and build up to them more. We know we spend a lot of our time working with food security surveys and we know our food security questions by heart. We can forget how weird they can sound to everyone else, especially over a chat. For our participants, it was the first time they’d seen something like this so they were at times confused about how to respond to the questions about their diet or their coping strategies. They were especially confused because the questions seemed to come out of nowhere, with no build up or putting them in context. By the second time around, they went through the survey much quicker, but we need to make sure to get the best first time responses. We need to speak normal language, not make everyone else try to speak our specialized jargon.

No one wants to interact with a robot: make the chatbot as chatty and friendly as possible. Our participants also advised us that it would be good to add some slang and colloquial language. But it is important to have it to feel like as natural an interaction as possible: As one of them said: “ If I want to say something or someone to talk to, I can write, and the chatbot can help and I can relax.’

Make it as intuitive as possible. The chabot users will have different backgrounds and tech literacy. Right now, as one of our participants put it, it’s accessible for “any educated person’, but we don’t want to limit our target audience. Our users might not even have secondary education so we want anyone who can use facebook to find it straightforward.img_2350

Make sure the bot recognises typos! Everyone knows how easy it is to make a typo on your smartphone so it’s essential that our chatbot recognises a few of the easy ones. When we ask people how many days in the past week that they ate vegetables for example, it’s pretty easy to give ‘3 days’ ‘three days’ ‘three’ and ‘3days’ and all mean the same thing! Even potentially typos like “theee days” or ‘three dyas”. We need to integrate these differences in text and typos as acceptable responses, asking for confirmation when needed, so we get the best results.

Put the bot on different messaging apps. One of the reasons why they were a bit hesitant with the bot at the beginning was the fact that they were not used to the Telegram app. It’s important for the bot to run on the app people use most. This can vary depending on the country, so when we do our pilot, we need to put the chatbot on the most commonly used messaging app.

Give people food price information for their areas. At the moment, our bot automatically reads the general WFP food price database. Whilst this is a cool way of looking at food prices all over the world, it’s not actually that useful on a day-to-day basis. Our participants said that knowing up-to-date regional prices would be great – as it would allow them to go to different parts of the country to buy food if the price was particularly low there. As we are already collecting high frequency price data, we want to be able to integrate this into the chatbot. This would be a great way to use the real-time data that we collect to give directly back to the people we collect from.

Overall the participants were positive to the bot as it stands, even saying that they think it’s ‘really cool’. However, there’s a lot of work to be done to make it more user friendly. Using this feedback we are going back to the drawing board. We hope to have an even better version for our official pilot in sub Saharan Africa later this year. We are very grateful to the people who helped us test this last week. They have much more pressing things to worry about so we thank them again for generously giving us a bit of time.

Leave a Reply

Your email address will not be published. Required fields are marked *