The main reasons for seeking an efficient feature selection are the machine learning algorithm get faster training, reduces the complexity of a model, facilitates interpretation and improves the accuracy of a model.

Look for Filter Methods , Wrapper Methods and Embedded Methods to learn more about your issue.

Filter methods are generally used as a preprocessing step. The selection of features is independent of any machine learning algorithms. Instead, features are selected on the basis of their scores in various statistical tests for their correlation with the outcome variable. Here you have to look for Linear discriminant analysis, Pearson’s Correlation, Chi-Square.

Some common examples of wrapper methods are :

Forward Selection: Is an iterative method in which we start with having no feature in the model. In each iteration, we keep adding the feature which best improves our model till an addition of a new variable does not improve the performance of the model.

Backward Elimination: Here, we start with all the features and removes the least significant feature at each iteration which improves the performance of the model. We repeat this until no improvement is observed on removal of features.

Recursive Feature elimination: It is a greedy optimization algorithm which aims to find the best performing feature subset. It repeatedly creates models and keeps aside the best or the worst performing feature at each iteration. It constructs the next model with the left features until all the features are exhausted. It then ranks the features based on the order of their elimination.

Embedded methods combine the qualities’ of filter and wrapper methods. It’s implemented by algorithms that have their own built-in feature selection methods.

Some of the most popular examples of these methods are LASSO and RIDGE regression which have inbuilt penalization functions to reduce overfitting.

Other example of embedded methods that could fits you is Regularized trees.

follow the link with some of these algorithms in sklearn.

sklearn - Feature Selection

I hope this could help you to start.

Thx for the complete answer Marcelo! Give me some time to try that out before validating it :) – Bertrand – 2019-01-04T09:45:20.413