I am deploying a sparse autoencoder.
Briefly, an autoencoder is a feedforward NN that is formed by a series of layers of decreasing dimension (the encoder), followed by a series of layers of increasing dimension (the decoder). The loss is computed by the MSE between NN’s input and decoder’s output. At the end of the encoder, instead, there is the encoded input.
I want to limit the number of activated neurons at the encoder’s ouput layer, in order to “simplify” the encoded data. But I am not sure that the steps I have followed are the right ones.
I thought to count the number of times in which each encoder’s output neuron was activated over the entire training dataset, compare such values with the desidered ones (an hyperparameter) and so the sparsity loss.
Then start fitting the model batch by batch, epoch by epoch, adding each time the sparsity loss to the training one, before the backpropagation step.
The sparsity loss is updated after each epoch.
Following this procedure, I have not obtained much satisfactory results. It seems that the sparsity term has not effects on the training. So before start looking for code bugs, I would like to have a confirm or a denial about the followed high-level procedure.