What is Sparse Autoencoder?
A Sparse Autoencoder is a type of autoencoder used to extract more features (i.e., to have more hidden nodes) than input nodes. It is also effective in preventing overfitting. Here is a simple structure of a Sparse Autoencoder.
To prevent overcomplete hidden layers, a Sparse Autoencoder limits or penalizes the loss function to avoid using all hidden nodes at once. That is, the autoencoder can use only a limited number of hidden nodes at once.
In the image above, the autoencoder can use only 2 nodes (white hidden nodes) at a time. The gray hidden nodes give very small values that they don’t affect the result.
However, not only the above 2 nodes are used in each step. For example, in the second step, another 3 nodes can be used.
In the third step, another 3 nodes can be used.
That is, sparse autoencoder uses a different node each time.