In Star Trek, the technology to instantly translate any language was first developed in the late 22nd century, but today researchers are already working on ways to take universal communication even further.
Mobile phone applications make it possible to translate into many different languages, but imagine giving a thumbs-up to someone from the Middle East or West Africa at the end of a chat – in most western cultures it is a sign of pleasure, but to them the signal is an outrageous insult.
‘It is often much easier to understand someone if you can see them because what is being said in a conversation extends far beyond the spoken word,’ said Professor Leo Wanner, a computational linguistics researcher at the Pompeu Fabra University in Barcelona, Spain.
‘It is about the tone of the voice, the facial expressions and the gestures. But there are social idiosyncrasies that vary between cultures.’
Prof. Wanner is leading the EU-funded KRISTINA project which aims to interpret language alongside cultural barriers in order to develop a personal translation assistant. They are working on software that can watch, listen and respond to migrants in ways that are socially appropriate to them or translate their needs for locals.
‘Helping people overcome barriers when they arrive in an unfamiliar country is not just about translating the language – it is rather about communication that requires social, cultural and background knowledge. This is what our socially intelligent conversation agent is trying to do,’ said Prof. Wanner.
‘It is about the tone of the voice, the facial expressions and the gestures. But there are social idiosyncrasies that vary between cultures.'
Professor Leo Wanner, Pompeu Fabra University, Spain
The agent takes the form of a digital platform that can respond to questions from users in their native language. Prof. Wanner said the goal is to develop a system that runs on devices like tablets and laptops.
The overall aim is to make these interactions as normal and natural as possible so users can ask questions, and chat, as if they were talking to another human being, which, for example, would help migrants integrate better when they arrive in a new country.
‘Migrants who arrive in European countries may not be familiar with the health system at all,’ he said. ‘Our agent would be able to assess their problem based on their age, location, gender, and other things – so it can tell them in natural language where they need to go.’
A prototype of the system, which can understand English, German, Spanish and Polish, is being evaluated in a healthcare context in Spain and Germany, while Turkish and Arabic will be added later.
In another setting, KRISTINA is being used to help Polish care workers who are looking after elderly patients in Germany.
‘The caregivers do not know the (German) language that well or the preferences of the elderly person they are looking after, so the agent can be an intermediary,’ said Prof. Wanner.
Around 4.1 % of the elderly population in Europe were born outside the EU. Interacting with people from such diverse backgrounds can be difficult and requires a deeper understanding of their background and cultures than most translation software can provide.
KRISTINA’s digital assistant aims to further help these people by recording sample conversations and building up a bank of basic knowledge. The system then builds on these conversations using machine learning, a type of automated data analysis, to help it recognise facial expressions, gestures and changes in the tone of voice.
Prof. Wanner said: ‘If we have an elderly person greet the virtual agent in a bit of a depressed voice in the morning, it will recognise that and ask what is wrong.
‘It will ask if they slept poorly or look for anything it can do to cheer them up. For example, it might remind them their family are coming to visit later that afternoon.’
The idea of communicating with people in ways that are familiar to them is something Professor Mirella Lapata, from the University of Edinburgh, UK, also believes is essential. She has been developing technology that doesn’t simply translate information for people – it makes it simpler.
Just think of the often incomprehensible terminology on tax forms and complex phrasing used on other official documents such as customs declarations. Then imagine tackling those in a second language.
Prof. Lapata, who is leading an EU European Research Council project called TransModal, is using machine learning to unravel key information from text, images, video and even computer code so it can be used by anyone.
‘What we are proposing is not just something like Google Translate but much more like Google Simplify,’ she explained. ‘The idea is you press a button or scan a document with your phone and get a simpler version.’
For someone with only a basic understanding of a second language, this could help them access information that has previously been mystifying to them, according to Prof. Lapata.
She said: ‘If you have an immigrant who comes to the UK, for example, they will struggle to do day-to-day stuff like going to the bank or to the post office because they cannot read the language. The idea is to simplify the information they see so they can understand it.’
Prof. Lapata hopes the algorithms involved could eventually be used through an application on a mobile phone or a button on a web-browser.
It is still early days, but Prof. Lapata believes the uses could also extend far beyond helping those who struggle with foreign languages.
‘People who have poor literacy skills could definitely benefit,’ she said. ‘Teachers could also use it to produce materials that are aimed at a range of abilities.’
If you liked this article, please consider sharing it on social media.
The big data explosion, which allows scientists to analyse factors such as people’s lifestyles, genes and medical records to develop personalised treatments for conditions, has so far mostly benefitted rare diseases with simple causes. But now, complex problems such as cardiovascular disease and dementia are getting the big data treatment.
An analysis of a newly cleaned-up dataset tracking Europe’s air pollution has revealed that nitrogen dioxide levels are on a steeper downward trend than previously thought, according to Dr Folkert Boersma from the Royal Netherlands Meteorological Institute, who says that ensuring the quality of Earth observation data can reveal new insights into climate change.
A sister and brother who created shock-activated protective gear featuring a starch liquid for people who in-line skate, motorcycle and do other risky sports, won one of the three first prizes at this year’s European Union Contest for Young Scientists (EUCYS).
Winners from Germany and Canada take home top prizes.
New observations may provide alternative explanations for dark energy.
We need to double-check the evidence on dark energy, as it may not exist at all, says Prof. Subir Sarkar.