Natural Language Processing - Natural language processing | NLP

There is a huge amount of textual information on the network, and NLP technology is needed to deal with these unstructured data.

This article introduces the basic concepts of NLP, 2 big tasks, 4 typical applications, and 6 practice steps.

To learn more about NLP-related content, please visit the NLP topic, and a 59-page NLP document download is available for free.

Visit the NLP topic and download a 59-page free PDF

 

Why is NLP important?

"Language understanding is the crown jewel in the field of artificial intelligence"

Bill Gates

Before the advent of artificial intelligence, machine intelligence processed structured data (such as data in Excel). But most of the data in the network is unstructured, such as: articles, pictures, audio, video...

In unstructured data, the amount of text is the most. Although he does not have much space for pictures and videos, his amount of information is the largest.

In order to be able to analyze and use these textual information, we need to use NLP technology to let the machine understand and use the text information.

 

What is natural language processing-NLP

Every animal has its own language, and the machine is also!

Natural Language Processing (NLP) is a bridge between machine language and human language to achieve the purpose of human-computer communication.

Humans communicate through language, and dogs communicate through barking. The machine also has its own way of communication, that is digital information.

Different languages ​​can't communicate. For example, humans can't understand dog barking. Even humans in different languages ​​can't communicate directly. They need translation to communicate.

This is especially true for computers. In order for computers to communicate with each other, people let all computers follow certain rules. These rules of computers are the language between computers.

Since there can be translations between different human languages, can humans and machines communicate directly through "translation"?

NLP is the bridge between humans and machines!

Why is "natural language" processing?

Natural language is the expression commonly used in daily life, which is what people usually mean by "speaking."

Natural language: I have a little camel on my back (unnatural language: my back is curved)

Natural language: The baby's agent sleeps baby's baby (a lot of this kind on Weibo)

 

NLP's 2 core mission

NLP has 2 core tasks:

1. Natural Language Understanding – NLU | NLI
2. Natural language generation-NLG

 

Natural Language Understanding – NLU|NLI

Natural language understanding means that the machine is like a human being, and has the ability to understand the language of a normal person. Because natural language has many difficulties in understanding (detailed below), NLU It is still far from being human.

5 difficulties in natural language understanding:

1. Language diversity
2. Language ambiguity
3. Language robustness
4. Language knowledge dependence
5. Language context

To learn more about NLU, check out this article.A text to understand natural language understanding - NLU (basic concept + practical application + 3 implementation)"

 

Natural language generation-NLG

NLG It is to cross the communication gap between humans and machines, and to convert non-verbal data into language formats that humans can understand, such as articles and reports.

NLG's 6 steps:

1. Content Determination – Content Determination
2. Text structure – Text Structuring
3. Sentence Aggregation – Sentence Aggregation
4. Grammaticalization – Lexicalisation
5. Reference Expression Generation – Referring Expression Generation|REG
6. Language Realization – Linguistic Realisation

To learn more about NLG, take a look at this article.A text to understand natural language generation – NLG (6 implementation steps + 3 typical applications)"

 

5 difficulties of NLP

1. Language is not regular, or the law is intricate.
2. Languages ​​can be freely combined and can be combined with complex language expressions.
3. Language is an open collection, and we can invent and create some new expressions.
4. Language needs to be linked to practical knowledge and has certain knowledge dependence.
5. The use of the language is based on the environment and context.

 

4 typical applications of NLP

emotion analysis

There are a lot of text messages on the Internet. The content of these messages is varied, but the emotions they express are the same: positive/positive-negative/negative.

Through sentiment analysis, you can quickly understand the user's grievances.

 

Chat robot

In the past, only Siri and Xiaobing robots were not very powerful, just as a way of entertainment. However, the rapid development of smart speakers in recent years has made everyone feel the value of chat robots.

And with the development of smart homes and smart cars in the future, chat bots will have greater use value.

 

Speech Recognition

Voice recognition has become a national quotation. Voice-to-text can be converted in WeChat, the destination can be directly spoken in the navigation of the car, and the elderly can use the input method to directly speak without learning pinyin...

 

machine translation

The current machine translation accuracy rate is already very high. You can use Google Translate to understand the meaning of the article. Traditional human flesh translation is likely to be unemployed in the future.

 

NNUM's 2 approach, 3 core steps

NLP can be handled using traditional machine learning methods or deep learning methods. Different approaches to 2 also correspond to different processing steps. Details are as follows:

Way 1: Traditional Machine Learning NLP Process

1. Corpus pretreatment
   1. Chinese corpus preprocessing 4 steps (detailed below)
   2. 6 steps for English corpus preprocessing (detailed below)
2. Feature engineering
   1. Feature extraction
   2. Feature selection
3. Select classifier

 

Way 2: NLP Process for Deep Learning

1. Corpus pretreatment
   1. Chinese corpus preprocessing 4 steps (detailed below)
   2. 6 steps for English corpus preprocessing (detailed below)
2. Design model
3. Model training

 

6 steps for English NLP corpus preprocessing

1. Word Segmentation – Tokenization
2. Stem extraction - Stemming
3. Lexical reduction – Lemmatization
4. Part of Speech Tagging – Parts of Speech
5. Named entity recognition-NER
6. Chunking-Chunking

 

4 steps for Chinese NLP corpus preprocessing

1. Chinese Word Segmentation – Chinese Word Segmentation
2. Part of Speech Tagging – Parts of Speech
3. Named entity recognition-NER
4. Remove stop words

 

Final Thoughts

Natural Language Processing (NLP) is a bridge between machine language and human language to achieve the purpose of human-computer communication.

 

NLP's 2 core tasks:

1. Natural Language Understanding – NLU
2. Natural language generation-NLG

 

5 Difficulties of NLP:

1. Language is not regular, or the law is intricate.
2. Languages ​​can be freely combined and can be combined with complex language expressions.
3. Language is an open collection, and we can invent and create some new expressions.
4. Language needs to be linked to practical knowledge and has certain knowledge dependence.
5. The use of the language is based on the environment and context.

 

Typical 4 applications for NLP:

1. emotion analysis
2. Chat robot
3. Speech Recognition
4. machine translation

 

NNUM 6 implementation steps:

1. Participle-Tokenization
2. Secondary extraction-stemming
3. Lexical reduction-lemmatization
4. Part of speech tagging - pos tags
5. Named entity recognition -Ner
6. Block-chunking

Baidu Encyclopedia Version + Wikipedia

 

 

Extended reading