How to determine feature importance in a neural network?



I have a neural network to solve a time series forecasting problem. It is a sequence-to-sequence neural network and currently it is trained on samples each with ten features. The performance of the model is average and I would like to investigate whether adding or removing features will improve the performance. I have constructed the neural network using keras.

The features I have included are:

  1. The historical data
  2. quarterly lagged series of the historical data (4 series)
  3. A series of the change in value each week
  4. Four time invariant features tiled to extend the length of the series. (another 4 series)

I am aware I could run the model many times changing the combination of features included each time. However, along with tuning the hyperparameters (for it might be that 8 features works really well with one set of hyperparameters but not with another set) this is really a lot of possible combinations.

Is there any separate way that I can use to guage if a feature is likely to add value to the model or not?

I am particuarly concerned that I have four time-invariant features being fed into the model which is designed to work with time varying data and I would like a way to measure their impact and if they add anything or not?


Posted 2019-01-27T14:01:53.513

Reputation: 298

1You basically want to assess the statistical significance of your features. There's no native inexpensive way to do with with a neural network. However, you could do this by fitting a separate regression with ARMA errors model and check coefficient p-values. – Digio – 2019-01-27T14:20:51.137

What you suggest sounds interesting, but could you expand upon your idea a little bit, I didn't fully understand. – Aesir – 2019-01-27T14:27:27.760

2You could try fitting a type of linear model to your series, using your neural network features as the dependent variables, then look at coefficient p-values to see which features have important impact to the series. There are many ways to do this, R has regression with ARMA errors (package forecast), python has the GLSAR class, and with some extra manual work you can do this using simple linear regression. – Digio – 2019-01-27T15:26:20.980



Don't remove a feature to find out its importance, but instead randomize or shuffle it.

Run the training 10 times, randomize a different feature column each time and then compare the performance. There is no need to tune hyper-parameters when done this way.

Here's the theory behind my suggestion: feature importance


Posted 2019-01-27T14:01:53.513

Reputation: 114

Do you mean by "randomize" setting the values of this feature to random numbers (between 0 and 1, depending on the scaling)? – NeStack – 2021-02-03T16:33:56.230


You can do this sort of thing using SHAP, it looks at permutation importance as well.

Brian Droncheff

Posted 2019-01-27T14:01:53.513

Reputation: 11