What are examples of approaches to dimensionality reduction of feature vectors?


Given a pre-trained CNN model, I extract feature vector of images in reference and query dataset with several thousands of elements.

I would like to apply some augmentation techniques to reduce the feature vector dimension to speed up cosine similarity/euclidean distance matrix calculation.

I have already come up with the following two methods in my literature review:

  1. Principal Component Analysis (PCA) + Whitening
  2. Locality Search Hashing (LSH)

Are there more approaches to perform dimensionality reduction of feature vectors? If so, what are the pros/cons of each perhaps?

Färid Alijani

Posted 2020-01-23T11:40:16.597

Reputation: 285



Dimensionality reduction could be achieved by using an Autoencoder Network, which learns a representation (or Encoding) for the input data. While training, the reduction side (Encoder) reduces the data to a lower-dimension and a reconstructing side (Decoder) tries to reconstruct the original input from the intermediate reduced encoding.

You could assign the encoder layer output ($L_i$) to a desired dimension (lower than that of the input). Once trained, $L_i$ could be used as a alternative representation of your input data in a lower feature-space, and can be used for further computations.

Autoencoder Architecture


Posted 2020-01-23T11:40:16.597

Reputation: 335

1Maybe this answer could be further improved if you link to a paper or implementation that shows the application of AE to reduce the dimensionality of feature vectors. If you consider images feature vectors, then, in a way, AE are commonly applied to reduce the dimensionality of images (or feature vectors), but what if the inputs are not images? – nbro – 2020-01-26T23:28:10.927

Is there any python library perform autoencoder efficiently? – Färid Alijani – 2020-02-03T06:44:42.500


@FäridAlijani Not that I know of. However, designing one in Keras wouldn't be much of a task. The following Keras blog might help : https://blog.keras.io/building-autoencoders-in-keras.html

– SoumyadeepB – 2020-02-03T21:18:47.780


Some examples of dimensionality reduction techniques:

Linear methods

  • PCA
  • CCA
  • ICA
  • SVD
  • LDA
  • NMF

Non-linear methods

  • Kernel PCA
  • GDA
  • Autoencoders
  • t-SNE
  • UMAP
  • MVU

Graph-based methods ("Network embedding")

  • Diffusion maps
  • Graph Autoencoders
  • Graph-based kernel PCA
    • Isomap
    • LLE
    • Hessian LLE
    • Laplacian Eigenmaps

Though there are many more.


Posted 2020-01-23T11:40:16.597

Reputation: 423