First, learn about ensemble learning and Boosting algorithms
Integrated learning belongs to machine learning. He is a "training idea" and is not a specific method or algorithm.
In real life, everyone knows that "people are more powerful," and "3 stinkers are the top ones." The core idea of integrated learning is that "people are more powerful." Instead of creating new algorithms, they combine existing algorithms to achieve better results.
Integrated learning picks up some simple basic models for assembly. The main ideas for assembling these basic models are 2 methods:
- Bagging (short for bootstrap aggregating, also known as "bagging method")
The Adaboost algorithm mentioned today is one of the Boosting algorithms.
Let's talk about Boosting algorithm again:
The core idea of Boosting is to pick the elite.
The most essential difference between Boosting and bagging is that he does not treat the basic model consistently. Instead, he tries to select the "elite" through constant testing and screening, and then gives the elite more voting rights. The poorly performing basic model gives Less voting rights, then combined with everyone's vote to get the final result.
most of the time,The result bias (bias) obtained by boosting is smaller.
- The basic model is linearly combined by the addition model.
- Each round of training improves the weight of the base model with a small error rate and reduces the weight of the model with a high error rate.
- Change the weight or probability distribution of the training data in each round, and reduce the weight of the sample by the weak classifier in the previous round, and reduce the weight of the previous round of the paired sample to make the classifier It has a good effect on misclassified data.
To learn more about integration algorithms or bagging algorithms, check out this article "An article to understand integrated learning (detailed bagging, boosting and their 4 point difference)"
What is the AdaBoost algorithm?
Boosting is a collection technique that attempts to create a strong classifier from many weak classifiers. This is done by trying to build a model from the training data and then creating a second model to try to correct the error from the first model. Add a model until you perfectly predict the training set or add the largest number of models.
AdaBoost is the first truly successful enhancement algorithm developed for binary classification. This is the best starting point for understanding help. The modern boost method is based on AdaBoost, the most famous of which is the random gradient enhancement machine.
AdaBoost is used for short decision trees. After the first tree is created, the performance of the tree on each training instance for the next tree to be weighted should focus on the attention of each training instance. Unpredictable training data is given more weight, while instances that are easier to predict are given less weight. The models are created one after the other, each model updating the weights on the training instances that affect the learning performed by the next tree in the sequence. After all the trees have been constructed, the new data is predicted and the performance of each tree is weighted according to the accuracy of the training data.
Because the algorithm is so focused on correcting errors, clean data with outliers must be removed.
7 pros and cons of Adaboost
AdaBoost algorithm advantages:
- Very good use of weak classifiers for cascading;
- Different classification algorithms can be used as weak classifiers;
- AdaBoost has a high degree of precision;
- Relative to the bagging algorithm andRandom ForestAlgorithm, AdaBoost fully considers the weight of each classifier;
Adaboost algorithm disadvantages:
- The number of AdaBoost iterations is also a poorly set number of weak classifiers, which can be determined using cross-validation;
- Data imbalance leads to a decrease in classification accuracy;
- Training is time consuming, and it is best to cut the point at each reselection of the current classifier;