Author Archive

Classification features are an important class of features.Classification features are discrete and non-continuous.

This article will introduce 5 mainstream coding methods for small and large classifications.And their respective advantages and disadvantages.

What are classification (category) characteristics?

Categorical features are used to represent classification. Unlike numerical features, which are continuous, categorical features are discrete.

such as:

• 性别
• city
• Colour
• IP address
• User's account ID

Some classification features are also numerical values, such as account ID and IP address.But these values ​​are not continuous.

Continuous numbers are numerical features, and discrete numbers are categorical features.

For continuous and discrete explanations, you can read this article: "Understanding of continuous and discrete"

Encoding of small classification features

Natural Number Encoding/Sequence Encoding-Ordinal Encoding

Certain classifications have a certain order, in this case, simple natural number coding can be used.

For example degree:

Bachelor-0

Master-1

Ph.D-2

One-Hot Encoding-One-Hot Encoding

For city, color, brand, material... these features are not suitable for coding with natural numbers, because these features have no ordering relationship.

The use of one-hot encoding can make different categories in an "equal position", and will not affect the classification because of the magnitude of the value.

For example, color classification (assuming there are only 3 colors):

Red-100

Yellow -010

Blue-001

Similar to one-hot encoding, there are "Dummy Encoding" and "Effect Encoding".

The implementation is similar, but there are some slight differences, and it is applicable to different scenarios.

Those who are interested can read this article:

'The difference between dummy variables and one-hot encoding"

'Assignment method: effect coding"

Encoding of large-scale classification features

Target Encoding

Target encoding is a very effective method to represent classification columns, and it only occupies a feature space, also known as mean encoding.Each value in this column is replaced by the average target value for that category.This can more directly express the relationship between categorical variables and target variables.

Extended reading of the target code: "Introduction to Target Encoding"

Hash encoding

The hash function is also a hash function that everyone often hears.The hash function is a deterministic function that maps a potentially unbounded integer to a finite integer range [1, m].

If there is a category with 1 values, if one-hot encoding is used, the encoding will be very long.With the use of hash encoding, no matter how many different values ​​there are in the classification, it will be converted into a fixed-length encoding.

Bin-Counting

The thinking of bin counting is a bit complicated: instead of using the value of a categorical variable as a feature, he uses the conditional probability of the target variable taking this value.

In other words, we do not encode the value of the categorical variable, but calculate the correlation statistics between the value of the categorical variable and the target variable to be predicted.

Summary of the advantages and disadvantages of different encodings

One-Hot Encoding-One-Hot Encoding

advantage:

1. easy to accomplish
2. The classification is very precise
3. Can be used for online learning

Things to note:

1. Inefficient calculation
2. Unable to adapt to growthable categories
3. Only applicable to linear models
4. For large data sets, large-scale distributed optimization is required

Hash encoding

advantage:

1. easy to accomplish
2. Model training costs are lower
3. Easy to adapt to new categories
4. Easy to handle rare types
5. Can be used for online learning

Things to note:

1. Only suitable for linear models or kernel methods
2. Unexplainable features after hashing
3. Accuracy is difficult to guarantee

Bin-Counting

advantage:

1. Minimal computational burden during training
2. Can be used for tree-based models
3. Easy to adapt to new categories
4. Use back-off method or minimum count graph to handle rare classes
5. Explainable

Things to note:

1. Need historical data
2. Need to delay update, not completely suitable for online learning
3. Very likely to cause a data breach

The above content is taken from: "Proficient in feature engineering"

Final Thoughts

Categorical features are discrete features, and numerical features are continuous.

For small classifications, commonly used encoding methods are:

1. Natural Number Encoding/Sequence Encoding-Ordinal Encoding
2. One-Hot Encoding-One-Hot Encoding
3. Dummy Encoding-Dummy Encoding
4. Effect Encoding-Effect Encoding

For large classifications, commonly used coding methods are:

1. Target Encoding
2. Hash encoding
3. Bin-Counting

Recommended articles:

'Machine learning category feature processing"

'Feature Engineering (XNUMX): Category Features"

Numerical features are the most common feature type, and numerical values ​​can be directly fed to the algorithm.
In order to improve the effect, we need to do some processing on numerical features. This article introduces 4 common processing methods: missing value processing, binarization, bucketing, and scaling.

What is a numerical feature?

Numerical features are features that can be actually measured.E.g:

• Human height, weight, three-dimensional
• The number of visits to the product, the number of times it was added to the shopping cart, and the final sales volume
• How many new users and returning users among the logged-in users

The features of the numerical class can be directly fed to the algorithm, why do we need to deal with it?

Because good numerical features can not only show the information hidden in the data, but also consistent with the model's assumptions.A good effect can be improved through proper numerical transformation.

For example, linear regression and logistic regression are very sensitive to the size of the value, so it needs to be scaled.

For numerical features, we mainly focus on 2 points:

1. 大小
2. distributed

The four processing methods mentioned below are optimized around size and distribution.

4 common processing methods for numerical features

1. Missing value processing
2. Binarization
3. Divide buckets/bins
4. Zoom

Missing value processing

In actual problems, we often encounter data missing.Missing values ​​will have a greater impact on performance.So it needs to be dealt with according to the actual situation.

There are three commonly used processing methods for missing values:

1. Fill in missing values ​​(mean, median, model prediction...)
2. Delete rows with missing values
3. Ignore it directly, and feed the missing value as part of the feature to the model for learning

Binarization

This processing method is usually used in counting scenarios, such as: the number of visits, the number of times a song has been listened to...

Example:

Predict which songs are more popular based on the user’s listening music data.

Assuming that most people listen to songs very averagely and will listen to new songs continuously, but there is a user who plays the same song 24 hours a day, and this song is very partial, resulting in a particularly high total number of listening to this song .If the total number of listening times is used to feed the model, it will mislead the model.At this time, you need to use "binarization".

The same user has listened to the same song N times, and only counts 1, so that everyone can find songs that everyone likes to recommend.

Divide buckets/bins

Take the income of each person as an example. The income of most people is not high, and the income of a very small number of people is extremely high and the distribution is very uneven.Some have a monthly income of 3000, and some have a monthly income of 30, which is several orders of magnitude.

This feature is very unfriendly to the model.This situation can be handled by bucketing.Bucketing is to divide numerical features into different intervals, and treat each interval as a whole.

Common bucketing:

1. age distribution
2. Commodity price distribution
3. Income distribution

Commonly used bucketing methods:

1. 固定数值的分桶（例如年龄分布：0-12岁、13-17岁、18-24岁…）、
2. Quantiles and buckets (for example, the price range recommended by Taobao: 30% of users choose the cheapest price range, 60% of users choose the medium price range, and 9% of users choose the most expensive price range)
3. Use the model to find the best bucket

Zoom

Linear regression and logistic regression are very sensitive to the magnitude of the value, and the large difference between different feature scales will seriously affect the effect.Therefore, the values ​​of different magnitudes need to be normalized.Scale different orders of magnitude into the same static range (for example: 0~1, -1~1).

Commonly used normalization methods:

1. z-score normalization
2. min-max standardization
3. Row normalization
4. Variance scaling

Extended reading:

'Data scaling: standardization and normalization"

'106-Data scaling (standardization, normalization) those things"