Recurrent Neural Network-Recurrent Neural Network | RNN

Convolutional Neural Network-CNN Already powerful, why do you still need RNN?

This article will explain the unique value of RNN in a simple and understandable way-processing sequence data. It will also explain some of the shortcomings of RNN and its variant algorithms.

Finally, I will introduce the practical application value and usage scenarios of RNN.

 

Why is RNN needed? What is the unique value?

Convolutional Neural Network – CNN Most of the ordinary algorithms have a one-to-one correspondence between input and output, that is, one input gets one output. There is no connection between different inputs.

But in some scenarios, one input is not enough!

In order to fill in the blanks below, it is not appropriate to take any of the previous words. We need to know not only all the words in front, but also the order between words.

This kind of scenario that needs to process "sequence data-a series of interdependent data streams" needs to be solved by using RNN.

Typical centralized sequence data:

1. Text in the article
2. Audio content in speech
3. Price trends in the stock market
4. ……

The reason why RNN can effectively process sequence data is mainly based on its special operating principle. Let us introduce the basic operating principle of RNN.

 

The basic principle of RNN

The structure of traditional neural networks is relatively simple: input layer-hidden layer-output layer.As shown below:

The biggest difference between RNN and traditional neural networks is that each time the previous output is brought to the next hidden layer and trained together. As shown below:

Let's use a specific case to see how RNN works:

If we need to judge the user's intention to speak (asking the weather, asking the time, setting an alarm...), the user says "what time is it?" We need to segment this sentence first:

Then input the RNN in order, we first use "what" as the input of the RNN and get the output "01"

Then, we input "time" to the RNN network in order, and get output "02".

In this process, we can see that when "time" is entered,The previous "what" output also had an effect (half of the hidden layer was black).

By analogy, all the previous inputs have an impact on the future output. You can see that the circular hidden layer contains all the previous colors. As shown below:

When we judge the intention, we only need the output "05" of the last layer, as shown in the figure below:

The disadvantages of RNN are also obvious

Through the above example, we have found that short-term memory effects are large (such as orange areas), but long-term memory effects are small (such as black and green areas). This is the short-term memory problem of RNN.

1. RNN has short-term memory problems and cannot handle very long input sequences
2. Training RNNs requires significant costs

Due to the short-term memory problem of RNN, later RNN-based optimization algorithms appeared. Let me give you a brief introduction below.

 

RNN's optimization algorithm

RNN to LSTM-Long and Short Term Memory Network

RNN is a kind of rigid logic. The later the input is, the greater the influence, the earlier the input is, the smaller the influence is, and this logic cannot be changed.

LSTM The biggest change that has been made is to break this rigid logic and instead use a set of flexible logic-only retaining important information.

To put it simply: Grasp the point!

For example, let's quickly read the following paragraph:

When we finish reading quickly, we may only remember the following important points:

LSTM is similar to the key points above,He can keep "important information" in long series of data and ignore unimportant information. This solves the problem of short-term memory of RNN.

The specific technical implementation principles are not expanded here. If you are interested, you can see the detailed introduction of LSTM.Long Short-Term Memory Network-LSTM"

 

From LSTM to GRU

Gated Recurrent Unit-GRU is a variant of LSTM.He retains the characteristics of LSTM to focus on and forget unimportant information, and it will not be lost during long-term propagation.

GRU mainly makes some simplifications and adjustments on the LSTM model, which can save a lot of time when the training data set is relatively large.

 

Applications and usage scenarios of RNN

As long as it involves the processing of sequence data, it can be used.NLP Is a typical application scenario.

Text generation: Similar to the fill-in-the-blank questions above, give context and then predict what the words in the space are.

machine translation: Translation work is also a typical sequence problem. The order of words directly affects the translation result.

Speech Recognition: Determine what the corresponding text is based on the input audio.

Generate image description: Similar to talking and looking at pictures. Give a picture to describe the content in the picture. This is often a combination of RNN and CNN.

Video tagging: He breaks down the video into pictures, and then uses image descriptions to describe the content of the pictures.

 

Summary

The unique value of RNN is that it can effectively process sequence data.For example: article content, voice audio, stock price trends...

The reason he can process sequence data is because the previous input in the sequence will also affect the subsequent output, which is equivalent to having a "memory function". However, RNN has serious short-term memory problems, and the long-term data impact is small (even if it is important information).

So based on RNN, there are variant algorithms such as LSTM and GRU. These variant algorithms have several characteristics:

1. Long-term information can be effectively retained
2. Select important information to keep, and choose "forgotten" for non-important information

Several typical applications of RNN are as follows:

1. Text generation
2. Speech Recognition
3. machine translation
4. Generate image description
5. Video tagging

 

Baidu Encyclopedia + Wikipedia

 

Further reading