First, learn about ensemble learning and Boosting algorithms

Integrated learning belongs to machine learning. He is a "training idea" and is not a specific method or algorithm.

In real life, everyone knows that "people are more powerful," and "3 stinkers are the top ones." The core idea of ​​integrated learning is that "people are more powerful." Instead of creating new algorithms, they combine existing algorithms to achieve better results.

Integrated learning picks up some simple basic models for assembly. The main ideas for assembling these basic models are 2 methods:

  1. Bagging (short for bootstrap aggregating, also known as "bagging method")
  2. boosting


The Adaboost algorithm mentioned today is one of the Boosting algorithms.

Let's talk about Boosting algorithm again:

Boosting core ideas

The core idea of ​​Boosting is to pick the elite.

The most essential difference between Boosting and bagging is that he does not treat the basic model consistently. Instead, he tries to select the "elite" through constant testing and screening, and then gives the elite more voting rights. The poorly performing basic model gives Less voting rights, then combined with everyone's vote to get the final result.

most of the time,The result bias (bias) obtained by boosting is smaller.

Boosting specific process

Specific process:

  1. The basic model is linearly combined by the addition model.
  2. Each round of training improves the weight of the base model with a small error rate and reduces the weight of the model with a high error rate.
  3. Change the weight or probability distribution of the training data in each round, and reduce the weight of the sample by the weak classifier in the previous round, and reduce the weight of the previous round of the paired sample to make the classifier It has a good effect on misclassified data.

To learn more about integration algorithms or bagging algorithms, check out this article "An article to understand integrated learning (detailed bagging, boosting and their 4 point difference)"


What is the AdaBoost algorithm?

Boosting is a collection technique that attempts to create a strong classifier from many weak classifiers. This is done by trying to build a model from the training data and then creating a second model to try to correct the error from the first model. Add a model until you perfectly predict the training set or add the largest number of models.

AdaBoost is the first truly successful enhancement algorithm developed for binary classification. This is the best starting point for understanding help. The modern boost method is based on AdaBoost, the most famous of which is the random gradient enhancement machine.

AdaBoost is used for short decision trees. After the first tree is created, the performance of the tree on each training instance for the next tree to be weighted should focus on the attention of each training instance. Unpredictable training data is given more weight, while instances that are easier to predict are given less weight. The models are created one after the other, each model updating the weights on the training instances that affect the learning performed by the next tree in the sequence. After all the trees have been constructed, the new data is predicted and the performance of each tree is weighted according to the accuracy of the training data.

Because the algorithm is so focused on correcting errors, clean data with outliers must be removed.


7 pros and cons of Adaboost

AdaBoost algorithm advantages:

  • Very good use of weak classifiers for cascading;
  • Different classification algorithms can be used as weak classifiers;
  • AdaBoost has a high degree of precision;
  • Relative to the bagging algorithm andRandom ForestAlgorithm, AdaBoost fully considers the weight of each classifier;

Adaboost algorithm disadvantages:

  • The number of AdaBoost iterations is also a poorly set number of weak classifiers, which can be determined using cross-validation;
  • Data imbalance leads to a decrease in classification accuracy;
  • Training is time consuming, and it is best to cut the point at each reselection of the current classifier;


Baidu Encyclopedia and Wikipedia

Baidu Encyclopedia version

Adaboost is an iterative algorithm whose core idea is to train different classifiers (weak classifiers) for the same training set, and then combine these weak classifiers to form a stronger final classifier (strong classifier).

Read More

Wikipedia version

AdaBoost is an acronym for Adaptive Boosting and is powered by Yoav Freund and Robert The machine learning meta-algory produced by Schapire, who won the 2003 Gödel Award for their work. It can be used in conjunction with many other types of learning algorithms to improve performance. The outputs of other learning algorithms ("weak learners") are combined into a weighted sum that represents the final output of the boosting classifier.

AdaBoost is adaptive in a sense that subsequent weak learners are adjusted to support instances that are misclassified by previous classifiers. AdaBoost is sensitive to noise data and outliers. In some problems, it may be less susceptible to overfitting problems than other learning algorithms. Individual learners may be weak, but as long as each learner's performance is slightly better than random guessing, the final model can prove to be integrated into powerful learners.

Read More