In Star Trek, the technology to instantly translate any language was first developed in the late 22nd century, but today researchers are already working on ways to take universal communication even further.
Mobile phone applications make it possible to translate into many different languages, but imagine giving a thumbs-up to someone from the Middle East or West Africa at the end of a chat – in most western cultures it is a sign of pleasure, but to them the signal is an outrageous insult.
‘It is often much easier to understand someone if you can see them because what is being said in a conversation extends far beyond the spoken word,’ said Professor Leo Wanner, a computational linguistics researcher at the Pompeu Fabra University in Barcelona, Spain.
‘It is about the tone of the voice, the facial expressions and the gestures. But there are social idiosyncrasies that vary between cultures.’
Prof. Wanner is leading the EU-funded KRISTINA project which aims to interpret language alongside cultural barriers in order to develop a personal translation assistant. They are working on software that can watch, listen and respond to migrants in ways that are socially appropriate to them or translate their needs for locals.
‘Helping people overcome barriers when they arrive in an unfamiliar country is not just about translating the language – it is rather about communication that requires social, cultural and background knowledge. This is what our socially intelligent conversation agent is trying to do,’ said Prof. Wanner.
‘It is about the tone of the voice, the facial expressions and the gestures. But there are social idiosyncrasies that vary between cultures.'
Professor Leo Wanner, Pompeu Fabra University, Spain
The agent takes the form of a digital platform that can respond to questions from users in their native language. Prof. Wanner said the goal is to develop a system that runs on devices like tablets and laptops.
The overall aim is to make these interactions as normal and natural as possible so users can ask questions, and chat, as if they were talking to another human being, which, for example, would help migrants integrate better when they arrive in a new country.
‘Migrants who arrive in European countries may not be familiar with the health system at all,’ he said. ‘Our agent would be able to assess their problem based on their age, location, gender, and other things – so it can tell them in natural language where they need to go.’
A prototype of the system, which can understand English, German, Spanish and Polish, is being evaluated in a healthcare context in Spain and Germany, while Turkish and Arabic will be added later.
In another setting, KRISTINA is being used to help Polish care workers who are looking after elderly patients in Germany.
‘The caregivers do not know the (German) language that well or the preferences of the elderly person they are looking after, so the agent can be an intermediary,’ said Prof. Wanner.
Around 4.1 % of the elderly population in Europe were born outside the EU. Interacting with people from such diverse backgrounds can be difficult and requires a deeper understanding of their background and cultures than most translation software can provide.
KRISTINA’s digital assistant aims to further help these people by recording sample conversations and building up a bank of basic knowledge. The system then builds on these conversations using machine learning, a type of automated data analysis, to help it recognise facial expressions, gestures and changes in the tone of voice.
Prof. Wanner said: ‘If we have an elderly person greet the virtual agent in a bit of a depressed voice in the morning, it will recognise that and ask what is wrong.
‘It will ask if they slept poorly or look for anything it can do to cheer them up. For example, it might remind them their family are coming to visit later that afternoon.’
The idea of communicating with people in ways that are familiar to them is something Professor Mirella Lapata, from the University of Edinburgh, UK, also believes is essential. She has been developing technology that doesn’t simply translate information for people – it makes it simpler.
Just think of the often incomprehensible terminology on tax forms and complex phrasing used on other official documents such as customs declarations. Then imagine tackling those in a second language.
Prof. Lapata, who is leading an EU European Research Council project called TransModal, is using machine learning to unravel key information from text, images, video and even computer code so it can be used by anyone.
‘What we are proposing is not just something like Google Translate but much more like Google Simplify,’ she explained. ‘The idea is you press a button or scan a document with your phone and get a simpler version.’
For someone with only a basic understanding of a second language, this could help them access information that has previously been mystifying to them, according to Prof. Lapata.
She said: ‘If you have an immigrant who comes to the UK, for example, they will struggle to do day-to-day stuff like going to the bank or to the post office because they cannot read the language. The idea is to simplify the information they see so they can understand it.’
Prof. Lapata hopes the algorithms involved could eventually be used through an application on a mobile phone or a button on a web-browser.
It is still early days, but Prof. Lapata believes the uses could also extend far beyond helping those who struggle with foreign languages.
‘People who have poor literacy skills could definitely benefit,’ she said. ‘Teachers could also use it to produce materials that are aimed at a range of abilities.’
If you liked this article, please consider sharing it on social media.
Imagine controlling your computer just by thinking. It sounds far-out, but real advances are happening on these so-called brain-computer interfaces. More researchers and companies are moving into the area. Yet major challenges remain, from user training to the reality of invasive brain implant procedures.
Artificial intelligence is growing ever more powerful and entering people’s daily lives, yet often we don’t know what goes on inside these systems. Their non-transparency could fuel practical problems, or even racism, which is why researchers increasingly want to open this ‘black box’ and make AI explainable.
In the summer of 2014 a strange building began to take shape just outside MoMA PS1, a contemporary art centre in New York City. It looked like someone had started building an igloo and then got carried away, so that the ice-white bricks rose into huge towers. It was a captivating sight, but the truly impressive thing about this building was not so much its looks but the fact that it had been grown.
Bilingual people can effortlessly switch between languages during everyday interactions. But beyond its usefulness in communication, being bilingual could affect how the brain works and enhance certain abilities. Studies into this could inform techniques for learning languages and other skills.
Live mycelium networks, capable of information processing, could be used as building materials.
Researchers are investigating whether bilingualism enhances certain cognitive abilities.
Dr Kate Rychert studies ocean plate structures.