This article is reproduced from the public intelligence expert,Original address

The 2018 year is over and it is time to start predicting 2019 deep learning. Here are my previous predictions and reviews of 2017 years and 2018 years:

About 2017 Year Forecast and Review. 2017's predictions cover hardware acceleration, convolutional neural networks (CNNThe dominant position, meta-learning, reinforcement learning, confrontational learning, unsupervised learning, migration learning, and as part of deep learning (DL), design patterns and experiments beyond theory.

prediction:

https://medium.com/intuitionmachine/10-deep-learning-trends-and-predictions-for-2017-f28ca0666669

review:

https://medium.com/intuitionmachine/10-deep-learning-trends-and-predictions-for-2017-f28ca0666669

About 2018 Year Forecast and Review. 2018's projections cover hardware startups, meta-learning instead of SGD, generation models, self-games, semantic gaps, interpretability, massive data research, teaching environments, conversational cognition, and artificial intelligence ethics.

prediction:

https://medium.com/intuitionmachine/10-fearless-predictions-for-deep-learning-in-2018-bc74a88b11d9

review:

https://medium.com/intuitionmachine/2018-retrospective-on-deep-learning-predictions-1cea68825bb3

By reviewing my predictions, I found that I was too optimistic and overestimated the speed of technological development. In general, it has always been in an exaggerated state of expectation. In hindsight, it is because the potential complexity of general cognition is neglected. We must now lower our expectations and focus on promising areas. These promising areas will gradually progress rather than “moon shots” (note: a crazy idea or a project that is unlikely to be realized).

Revolutionary progress should occur in stages, and what we are experiencing today is the main obstacle to achieving the Interventional level. This does not mean that we can't make any progress, but there are many unresolved results in the current maturity level, and these results are ready for development. The progress of DL in 2019 year will mainly focus on this pragmatic understanding.

Here are my predictions, as they were in previous years, as a guide to tracking DL progress.

1. Deep learning hardware acceleration slows down

Deep learning hardware acceleration has slowed down, and pulsating arrays have brought tremendous acceleration to the world in 2017 years. We can't expect a significant increase in 2019's computing power. NVidia's Turing core is only a little faster than the Volta core. Google's TPUv3 system is now liquid-cooled and has a higher density than previous products. I don't think there will be any major architectural improvements in 2019, so don't increase as much as you did in previous years.

However, we will see that GraphCore and Gyrfalcon's new architecture circumvents the power cost of memory transfers and supports sparse operations, but needs to change the deep learning format to accommodate these new architectures, as well as the need for new hardware research, inspired by biology. Nano-intentionality.

2. Unsupervised learning has been resolved, but not expected

The way of thinking without unsupervised learning is wrong. LeCun's cake theory is wrong, and different types of learning relationships should look like this:

Why is UL the lowest value and the least difficult? That's because there is no goal, you can do any cluster that may be valid or invalid. Ultimately, it comes down to the performance of higher layers based on UL embedding. UL embedding is essentially rich in a priori data, and how these a priori are utilized depends on the upstream process with the target. ELMO andBERTIt has been found that we can train the UL used to predict (or generate) its data, which is a good basis for upstream tasks. UL is basically supervised learning, and its tags already exist in the data. In short, UL has been solved, but not as expected by most practitioners. If the network can make good predictions or can generate good faxes of raw data, then this is UL.

Therefore, everyone believes that solving UL will be a major development because people can use data without human tags. Unfortunately, because the free stuff is easy to extract, it has been solved. My prediction for UL in 2019 is that researchers will accept this new perspective and instead focus on more valuable research (ie continual or interventional learning).

3. Meta-learning is only for research

Our understanding of meta-learning seems to be as vague as the understanding of unsupervised learning. Meta-learning practiced today is more like migration learning. In fact, more advanced meta-learning can build and improve your own models. Meta-learning should be able to build extrapolative and creative learning models, but we cannot achieve this capability.

Any learning method suitable for multiple domains is technically a meta-learning algorithm. For example, gradient descent, genetic algorithms, self-game, and evolution are all meta-learning algorithms. The goal of the meta-learning approach is to develop algorithms that are well learned in many fields.

There are few known meta-learning algorithms, but there is a meta-learning algorithm that we don't understand. We don't understand the meta-learning algorithms people use. In addition, meta-learning is a too common problem like unsupervised learning, so that it is impossible to understand how to solve it in a general way. Maybe there is really no free lunch in the world.

I think that some specific methods (such as generating models, hybrid models, and course training) will have a better chance of getting more valuable results, which means that the meta-learning algorithms we find are only useful for certain types of learning tasks. Just as learning through gradient descent only accelerates gradient descent for a particular task, meta-learning can only improve learning in the tasks it has seen. In short, meta-learning is at best interpolated and cannot be generalized. Perhaps there is no universal meta-learning method, and there is a set of meta-learning methods that can be pieced together to produce an effective curriculum.

In short, meta-learning will still need to be studied.

4. Application in science generates computational models

We will have better control over the build model. There are three types of generation models that have proven to be effective: variational self-encoders,GANAnd stream-based generation models. I hope to see the rapid development of the GAN and Flow models and the progress of VAE. I also expect to see this complex adaptive system application (ie weather, fluid simulation, chemistry and biology) in scientific exploration. Progress in this area will have a profound impact on scientific progress.

Application of 5. Hybrid Model in Forecasting

Deep learning has an advantage in providing high-dimensional system predictions. However, deep learning still cannot develop its own abstract model, which is still the basic obstacle to interpretative and extrapolative prediction. To complement these limitations, we will see a hybrid dual process solution that combines existing models with model-free learning.

I think using a hand-crafted model can alleviate the inefficiency of modelless RL. I am looking forward to the progress of the graph network, and when these graphs deviate from the previous model-based models, we will see impressive results. I also hope to improve predictive power by blending existing symbolic algorithms with DL.

The industrialization of DL will not be due to the progress we have made in migration learning, but through the integration of artificial models and DL training models.

6. More imitation learning methods

Imitation does not require extrapolation, so we will see considerable progress in mimicking various existing systems. In order to be able to mimic behavior, the machine only needs to create a descriptive model that reflects the behavior. This is easier than generating modeling because generating modeling must discover unknown generation constraints. The reason the build model works well is that all it does is mimic the data rather than inferring the potential causal model of the generated data.

7. More deep learning integration design exploration

We will see many of the research on generating models moved to existing design tools. It first appeared in the field of vision and gradually evolved in other directions.

In fact, we can even think of the progress of AlphaGo and AlphaZero as a design exploration. Competitive Go and Chess players have begun to explore the exploration strategies introduced from the DeepMind game AI to develop new strategies that have not been explored before.

The simple matching algorithm and scalability of deep learning methods will become brainstorming machines that can improve designs completed by humans. Many deep learning methods are now integrated in Adobe and AutoDesk products. Style2Paints is a great example of a deep learning method integrated with standard desktop applications.

Deep learning networks can reduce the cognitive load people need to complete tasks in a workflow. Deep learning allows the creation of tools that are good at handling more ambiguous and confusing cognitive details. These all need to reduce information overload, increase recall rates, extract text and make faster decisions.

8. End-to-end training attenuation, future development will focus on developmental learning

The rewards of end-to-end training will be reduced, and we will see networks trained in different environments to learn professional skills, stitching these methods together to form new ways to build blocks of more complex skills. I look forward to seeing the progress of course training in 2019 and I hope to see more research inspired by the development of human babies. Training the network to perform complex tasks will involve complex reward settings, so we need to improve the way to solve this problem.

9. Richer natural language processing embedding

NLPProgress has been made in 2018 years, mainly due to advances in creating word-embedded unsupervised learning methods, and 2018 NLP progress can be attributed to more advanced neural embedding (ELMO, BERT). By simply replacing the richer embedding, these embeddings improve many upstream NLP tasks, and the work in the graph network can be further enhanced.Deep learningNLP function.

TransformerThe network has proven to be very valuable in NLP and I hope it will continue to be used in other areas. I think the dominant position of the ConvNet network will be challenged by the Transformer network. My gut feeling is that attention is a more general mechanism for implementing invariance or covariance than the fixed mechanisms available to ConvNets.

10. Using cybernetics and system thinking

A major drawback of deep learning practices is the lack of understanding of the big picture. We are at a time when we need to draw inspiration from more non-traditional sources, which I believe are the study of previous cybernetics and its related systems of thinking. We need to start thinking about how to build a powerful intelligent infrastructure and intelligent expansion. This requires a machine learning mind that transcends many current researchers.

Michael commented in his article "Artificial Intelligence-The Revolution Has Not Happened" that Norbert Wiener's cybernetics has "dominated the current era". Cybernetics and systems thinking will help us develop a more comprehensive approach to designing AI systems, and successful AI deployment will ultimately align with how they are aligned with the needs of human users. This will require exploring and developing a holistic approach that integrates various interacting parts.

Many new and deep learning methods can be traced back to the ideas in cybernetics. The understanding that autonomous AI needs to include a subjective perspective in its world model will increase. Predictive coding, internal-to-external architecture, embodying learning, timely reasoning, intrinsic motivation, curiosity, self-modeling, and operational representation are all relevant in this paradigm.

Final Thoughts

Deep learning continues to make progress at a breakthrough speed, and I hope that research can be transformed into industrial applications. A common shortcoming in understanding deep learning in today's market is the inability to develop a holistic solution to existing problems. The ability to create a solution that integrates the DL as a component into the whole will be a sought-after skill. The machine learning methodology may be wrong, and we can find a more appropriate perspective in cybernetics. We may not be able to implement AGI in the short term, but the tools and methods available for deep learning can serve as a solid foundation for valuable applications in science and business.