Why isn't dimension sampling used with gradient boosting machines (GBM)?

7

1

GBMs, like random forests, build each tree on a different sample of the dataset and hence, going by the spirit of ensemble models, produce higher accuracies. However, I have not seen GBM being used with dimension sampling at every split of the tree like is common practice with random forests.

Are there some tests that show that dimensional sampling with GBM would decrease its accuracy because of which this is avoided, either in literature form or in practical experience?

Nitesh

Posted 2014-11-25T09:40:20.040

Reputation: 1 515

2What GBM implementations have you looked at? sklearn's GradientBoostingClassifier/Regressor have a max_features parameter and XGBoost has colsample_bylevel and colsample_bytree parameters that control how many features are sampled for each tree / split. – stmax – 2016-03-11T23:20:59.390

@stmax would you mind posting this comment as an answer? – Nitesh – 2016-03-29T16:17:44.690

Answers

2

sklearn's GradientBoostingClassifier / GradientBoostingRegressor have a max_features parameter and XGBoost has colsample_bylevel and colsample_bytree parameters that control how many features are sampled for each tree / split.

stmax

Posted 2014-11-25T09:40:20.040

Reputation: 1 536

0

I have never read (or tried) subsampling techniques on each node of the tree. I do not say that they might not exist, but it looks odd the statement "like is a common practice with random forests".

Other than that, subsampling or bootstrapping the sample for each tree from random forests gives stochastic gradient boosting Friedman, 1999. They give better results with subsampling than without and more details are given also in Elements of Statistical Learning, page 358, 10.12.2 Subsampling.

rapaio

Posted 2014-11-25T09:40:20.040

Reputation: 3 864

ESL - the Bible. – StephenBoesch – 2019-02-07T16:17:01.360