The existing model also "does not understand" natural language: 20 researchers talk about the four open issues of NLP

This article is reproduced from the public head of the Microsoft Research Institute AI headline,Original address

Deep Learning Indaba 2018 is a deep learning summit hosted by DeepMind and was held in Stellenbosch, South Africa during the 9 month of this year. At the meeting, AYLIEN research scientist Sebastian Ruder, DeepMind senior research scientist Stephan Gouws and Stellenbosch University lecturer Herman Kamper organized the Frontiers of Natural Language Processing session and interviewed more than 20NLPResearcher in the field. Not long ago, Sebastian Ruder published the interview record of the interview and the PPT of the conference speech, summarizing the entire interview. This article is based on expert interviews and panel discussions at the time, focusing on 4 major open issues in the NLP field:

• Natural language understanding
• Low resource language NLP
• Reasoning for large or multiple documents
• Data sets, questions and assessments

Natural language understanding

I think the biggest open question is about natural language understanding. We should develop a system that reads and understands text like humans by constructing textual representations. Until then, all our progress has been to improve the ability to match system patterns.

- Kevin Gimpel

In the expert interview, many experts believe that natural language understanding (NLUThe problem is at its core because it is a prerequisite for many tasks (such as natural language generation). They believe that the current model does not yet have a "real" understanding of natural language.

Inherent bias vs learning from scratch 

A key question is which bias terms and structures should be added to the model to get closer to natural language understanding. Many experts mentioned in the interview that the model should learn common sense. In addition, they have repeatedly mentioned the dialogue system (and chat bots).

On the other hand, with regard to reinforcement learning, David Silver believes that we will eventually want to let the model self-learn everything, including algorithms, features and predictions. Many experts hold the opposite view and believe that the understanding module should be embedded in the model.

Program synthesis

Omoju Miller thinks it's very difficult to embed understanding modules in a model. We don't know the mechanics behind NLU and how to evaluate them. She believes that we may be able to get inspiration from the synthesis of programs, based on advanced specifications to automatically learn the program. Such ideas are related to neural module networks and neural programmer-interpreters.

She also suggested that we should review the methods and frameworks developed in the 1980s and 1990s (such as FrameNet) and combine them with statistical methods. This should help us to infer the common sense attributes of the object, such as whether the car is a vehicle, whether the car has a handle, etc. Inferring this common sense knowledge is the focus of recent NLP data sets.

Embodied learning

Stephan Gouws believes that we should use information from structured data sources and knowledge bases such as Wikidata. He believes that human beings learn the language by applying their surroundings to the body through experience and interaction. One might think that there is a learning algorithm that can learn NLU from scratch when used in an information-rich environment with an appropriate reward structure. However, the amount of calculation for such an environment is enormous. AlphaGo requires a huge infrastructure to solve a well-defined board game. The creation of a general algorithm for continuous learning is related to lifelong learning and general problem solvers.

Many people think that since we are moving in the direction of physical learning, we should not underestimate the infrastructure and computing power needed to fully possess an agent. Therefore, waiting for a qualified physical learning language seems to be a fantasy. However, we can gradually approach this end point, such as grounded language learning in a simulated environment, using multimodal data learning, and so on.

emotion

Omoju believes that it is very difficult to integrate human emotions and other factors into the body. On the one hand, understanding emotions requires a deeper understanding of the language. On the other hand, we may not need an agent that truly has human emotions. Stephan said that the Turing test is defined as imitative and antisocial, and although there is no emotion, it can deceive humans and make people think that it has emotions. So we should try to find solutions that don't require physical and emotional, but they can understand human emotions and help people solve problems. Indeed, sensor-based emotion recognition systems are constantly improving, and text emotion detection systems have also made great progress.

Cognitive and neuroscience

At the meeting, there were audience questions and how much neuroscience and cognitive science knowledge we used in building the model. Neuroscience and cognitive science knowledge is an important source of inspiration and can be used as a guide to shaping thinking. For example, multiple models attempt to mimic human thinking. AI and neuroscience are complementary.

Omoju recommends that you draw inspiration from cognitive science theories such as Piaget and Vygotsky's cognitive development theory. She also urged everyone to conduct interdisciplinary research, which has resonated with other experts. For example, Felix Hill recommends everyone to attend a cognitive science conference.

NLP in low resource scenarios

Responding to less data scenarios (low resource languages, dialects, etc.) is not a completely "blank" problem, because there are already many promising ideas in the field, but we have not found a universal solution to solve such problems. .

- Karen Livescu

The second topic we explored was the generalization of areas beyond training data in low-resource scenarios. In the Indaba scene, a natural focus is on low resource languages. The first question focuses on whether it is necessary to develop a specialized NLP tool for a particular language, or whether a general NLP study is sufficient.

Universal language model

Bernardt believes that there is a common commonality between languages ​​that can be exploited through a common language model. The challenge then is how to get enough data and computing power to train such a language model. This is related to recent training across languagesTransformLanguage models are closely related to the study of cross-language sentence embedding.

Cross-linguistic representation

Stephan said that scholars studying low-resource language are not enough. In Africa alone, there are 1250-2100 languages, most of which are not of interest to the NLP community. Whether or not to develop a dedicated tool also depends on the type of NLP task to be processed. The main problem with existing models is their sample efficiency. Cross-language word embedding is very efficient at using samples because they only require translation pairs of words, even monologous data. They can align the word embedding space well to complete coarse-grained tasks such as topic categorization, but cannot complete fine-grained tasks such as machine translation. However, recent research suggests that these embeddings can create important building blocks for unsupervised machine learning.

On the other hand, complex models dealing with advanced tasks such as questions and answers require thousands of training samples to be learned. Moving tasks that require actual natural language understanding from high resource languages ​​to low resource languages ​​is still very challenging. Cross-language datasets with such tasks (such as XNLIThe development of a powerful cross-language model for more inference tasks should become easier.

Income and impact

Insufficient resources are essentially only a small amount of text available, and whether NLP's advantages are limited in this case is also a problem. Stephan showed strong disagreements. He reminded us that as practitioners of ML and NLP, we tend to look at issues in an informational way, such as maximizing the likelihood of data or improving benchmarks. To take a step back, the real reason we study the NLP problem is to build a system that overcomes barriers. We want to build models that allow people to browse non-native news and ask questions about health without being able to see a doctor...

Considering these potential effects, building a low-resource language system is actually one of the most important areas of research. Low resource languages ​​may not have much data, but there are so many languages. In fact, most people are talking about a resource-poor language. Therefore, we really need to find a way to let the system run under this setting.

Jade believes that our community is focused on languages ​​that have a lot of data, and it seems a bit ironic that these languages ​​are well educated around the world. What really needs our attention is the low-resource language that doesn't have much data available. The subtlety of Indaba is that people inside are advancing this low-resource language research and have made some progress. Given the scarcity of data, even a simple system like a word bag can have a major impact on the real world. Audience Etienne Barnard pointed out that he observed a different effect of speech processing in the real world: compared to the native language system, if the English system is suitable for the user's dialect, they are often more motivated to use the English system.

Motivation and skills

Another listener said that people are more motivated to do work with highly visual benchmarks, such as English-Chinese machine translation, but lack motivation in low-resource language. Stephan believes that the motivation is that the problem has not been resolved. However, the correct demographics do not have the skills needed to solve these problems. We should focus on teaching machine translation and other similar skills to help you get the ability to solve these problems. However, if cross-language benchmarking becomes more common, more progress will be made in the low-resource language arena.

Data accessibility

Jade finally mentioned that the lack of available data sets in low-resource languages ​​(such as some languages ​​in Africa) is a big problem. If we create a data set and make it very easy to get (such as putting it on openAFRICA), this will greatly motivate everyone and lower the entry barrier. Providing test data in multiple languages ​​is usually sufficient as it helps us evaluate cross-language models and track progress. Another data resource is the South African Centre for Digital Language Resources (SADiLaR), which contains many South African languages.

Reasoning for large text and multiple texts

Efficiently characterize large texts. Existing models are primarily based on cyclic neural networks that do not adequately characterize longer texts. Inspired by the graphRNNWorkflows have potential for development because they are easier to train than regular RNNs, although only limited improvements have been seen so far, and they have not been widely adopted.

- Isabelle Augenstein

Reasoning for large texts and multiple texts is also a big open question. The recent NarrativeQA dataset is a good benchmark example that fits this background. Using a large context for reasoning is closely related to the NLU, and it is necessary to greatly expand the existing system so that it can read the entire book or the entire movie script. There is a key question here: Do we need to train better models or just train on more data? The discussion is not discussed here.

Studies such as OpenAI Five show that if the amount of data and the amount of calculations are greatly increased, the tasks that the existing model can accomplish will be very impressive. With enough data, existing models can perform well in larger contexts. The problem is that data with a lot of text is very rare and the cost of acquisition is very expensive. Similar to language modeling and skip-thoughts, we can imagine a file-level unsupervised task that requires predicting the next paragraph or next chapter of a book, or deciding which chapter should be in the next chapter. However, this goal is likely to be too simple – inefficient to learn useful characterizations.

Developing a method that can more effectively characterize context and track relevant information while reading a file seems to be a more practical direction. Multi-file summaries and multi-file questions and answers are consistent with this research direction. Similarly, we can build models with improved memory capabilities and lifelong learning capabilities.

Data sets, questions and assessments

Perhaps the biggest problem is how to define the problem itself. Properly defined questions refer to building data sets and assessment steps to properly measure our progress on specific goals. If you can simplify all the problems into a Kaggle-style competition, things are much simpler!

- Mikel Artetxe

This article does not have a free space to discuss the current benchmark and evaluation settings, the relevant answers can refer to the survey results. The last question is what is the most urgent NLP problem in African society. The answer given by Jade is the lack of resources. It is very important to let people use the language to access all educational resources of interest in the language.