Untrained feature extractor with batch norm layers active

[ptrblck]

@SU801T

Would this be considered feature extraction with an “untrained” network?

Yes, I guess you could call DenseNetConv an untrained feature extractor.

@SU801T

If I allow this to train, for example by applying densenet.train() will this be sufficient in allowing batch normalisation statistics to be generated for each mini-batch as well as allow for the running means and standard deviations to be learnt and applied during inference, while keeping the convolutional layers untrained?

Yes. You’ve frozen all trainable parameters from the feature extractor by setting their requires_grad attribute to False, while the linear layers in MyDenseNetDens should still be trainable since no attributes were changed. Calling .train() on the model will normalize the forward activations using the batch stats and will update the running stats.