How to choose the features for a neural network?



I know that there is no a clear answer for this question, but let's suppose that I have a huge neural network, with a lot of data and I want to add a new feature in input. The "best" way would be to test the network with the new feature and see the results, but is there a method to test if the feature IS UNLIKELY helpful? Like correlation measures etc?


Posted 2014-07-10T10:07:13.523

Reputation: 1 597

1A non-random correlation might be an indicator that the feature is useful. But I'm not so sure about pre-training tests that could rule ideas out. The paper you link makes it clear that non-linear correlations are not well detected by the available tests, but a neural net has a chance of finding and using them. – Neil Slater – 2014-07-10T11:28:15.503



A very strong correlation between the new feature and an existing feature is a fairly good sign that the new feature provides little new information. A low correlation between the new feature and existing features is likely preferable.

A strong linear correlation between the new feature and the predicted variable is an good sign that a new feature will be valuable, but the absence of a high correlation is not necessary a sign of a poor feature, because neural networks are not restricted to linear combinations of variables.

If the new feature was manually constructed from a combination of existing features, consider leaving it out. The beauty of neural networks is that little feature engineering and preprocessing is required -- features are instead learned by intermediate layers. Whenever possible, prefer learning features to engineering them.

Madison May

Posted 2014-07-10T10:07:13.523

Reputation: 1 959

I always thought to compare the value to predict with the features, you are talking about correlation between features. Is your answer applicable also to my case? in theory I should add only new features that are correlated to the value to predict, right? – marcodena – 2014-07-10T19:06:57.880

That's also a valuable metric -- just updated my answer to address that as well. – Madison May – 2014-07-10T19:18:34.210

In short, strong correlations with the value to predict is a great sign, but weak correlation with the value to predict is not necessarily a bad sign. – Madison May – 2014-07-10T19:19:17.200

Thanks. I'm writing a report and I wanted to show the linear/non-linear correlations in order to justify the features (even before the results). Does it make any sense? From your answer I could make a matrix of correlations but maybe it's nosense – marcodena – 2014-07-10T19:27:33.073

A matrix of correlations would make more sense if you were performing linear regression, but it could also be a useful metric for neural nets. Give it a go and see what you get! – Madison May – 2014-07-10T21:29:06.873

1I would use non-linear correlations, but ok thanks – marcodena – 2014-07-10T23:17:47.063


If you are using sklearn, there is a good function available called model.feature_importances_. Give it a try with your model/new feature and see if it helps. Also look here and here for examples.


Posted 2014-07-10T10:07:13.523

Reputation: 89