First, learn about ensemble learning and Boosting algorithms
Ensemble learning belongs to machine learning. It is a "training idea", not a specific method or algorithm.
In real life, everyone knows that "many people have great power" and "three dreadful tanners put Zhuge Liang together." The core idea of ensemble learning is "many people have more power". It does not create new algorithms, but combines existing algorithms to get better results.
Integrated learning picks up some simple basic models for assembly. The main ideas for assembling these basic models are 2 methods:
- Bagging (short for bootstrap aggregating, also known as "bagging method")
The Adaboost algorithm mentioned today is one of the Boosting algorithms.
Let's talk about Boosting algorithm again:
The core idea of Boosting is to pick the elite.
The most essential difference between Boosting and bagging is that he does not treat the basic model uniformly, but selects the "elite" after constant testing and screening, and then gives the elite more voting rights. Fewer voting rights, then the votes of all people are combined to get the final result.
most of the time,The result bias (bias) obtained by boosting is smaller.
- The basic model is linearly combined by the addition model.
- Each round of training improves the weight of the base model with a small error rate and reduces the weight of the model with a high error rate.
- Change the weight or probability distribution of the training data in each round, and reduce the weight of the sample by the weak classifier in the previous round, and reduce the weight of the previous round of the paired sample to make the classifier It has a good effect on misclassified data.
To learn more about integration algorithms or bagging algorithms, check out this article "An article to understand integrated learning (detailed bagging, boosting and their 4 point difference)"
What is the AdaBoost algorithm?
Boosting is a collection technique that attempts to create a strong classifier from many weak classifiers. This is done by trying to build a model from the training data and then creating a second model to try to correct the error from the first model. Add a model until you perfectly predict the training set or add the largest number of models.
AdaBoost is the first truly successful enhancement algorithm developed for binary classification. This is the best starting point for understanding help. The modern boost method is based on AdaBoost, the most famous of which is the random gradient enhancement machine.
AdaBoost is used for short decision trees. After the first tree is created, the performance of the tree on each training instance for the next tree to be weighted should focus on the attention of each training instance. Unpredictable training data is given more weight, while instances that are easier to predict are given less weight. The models are created one after the other, each model updating the weights on the training instances that affect the learning performed by the next tree in the sequence. After all the trees have been constructed, the new data is predicted and the performance of each tree is weighted according to the accuracy of the training data.
Because the algorithm is so focused on correcting errors, clean data with outliers must be removed.
7 pros and cons of Adaboost
AdaBoost algorithm advantages:
- Very good use of weak classifiers for cascading;
- Different classification algorithms can be used as weak classifiers;
- AdaBoost has a high degree of precision;
- Relative to the bagging algorithm andRandom ForestAlgorithm, AdaBoost fully considers the weight of each classifier;
Adaboost algorithm disadvantages:
- The number of AdaBoost iterations is also a poorly set number of weak classifiers, which can be determined using cross-validation;
- Data imbalance leads to a decrease in classification accuracy;
- Training is time consuming, and it is best to cut the point at each reselection of the current classifier;
Baidu Encyclopedia and Wikipedia
Adaboost is an iterative algorithm whose core idea is to train different classifiers (weak classifiers) for the same training set, and then combine these weak classifiers to form a stronger final classifier (strong classifier).
AdaBoost is an acronym for Adaptive Boosting and is powered by Yoav Freund and Robert The machine learning meta-algory produced by Schapire, who won the 2003 Gödel Award for their work. It can be used in conjunction with many other types of learning algorithms to improve performance. The outputs of other learning algorithms ("weak learners") are combined into a weighted sum that represents the final output of the boosting classifier.
AdaBoost is adaptive in a sense that subsequent weak learners are adjusted to support instances that are misclassified by previous classifiers. AdaBoost is sensitive to noise data and outliers. In some problems, it may be less susceptible to overfitting problems than other learning algorithms. Individual learners may be weak, but as long as each learner's performance is slightly better than random guessing, the final model can prove to be integrated into powerful learners.
Among the advantages and disadvantages of AdaBoost, the right side of the picture should be the disadvantage of AdaBoost