Each layer of a neural network learns features of the input data. The first layer learns low-level features (e.g. edges in images). Each subsequent layer learns more abstract features. Then the features from the last hidden layer are combined for classification or regression.
How to make sure that the features learned by a neural network are not correlated?
I think that I can force a neural network to learn correlated features in the following way. Remove the output layer, so that the last hidden layer produces $n$ outputs. Then train this network on $n$ target variables that are not independent but are linear combinations of say two or three independent targets (maybe with some noise added to avoid true linear dependence). As neural networks are universal approximators, it should be possible to train such a network. However, in that case, all neurons in the final layer will be redundant except those two or three neurons that correspond to the linearly independent targets.
This redundancy can be present in each layer. How to avoid it? Of course, I can check the correlation of neuron outputs after the training, remove the correlated neurons, and retrain the network but it looks inefficient. How to make the network learn uncorrelated features from the beginning?
What comes to my mind is using different and independent (e.g. orthogonal) activation functions for each neuron within the layer. Are there such networks? Usually, activation functions are taken to be the same.