1

Let's scope this to just classification.

It's clear that if you fully grow out a decision tree with no regularization (e.g. max depth, pruning), it will overfit the training data and get full accuracy down to Bayes error*.

Is this universally true for all non-parametric methods?

*Assuming the model has access to the "right" features.

there is no known mathematical result that applies to "non-parametric models" in general. However it seems to be the case

– Nikos M. – 2020-07-16T12:10:52.137