I’ve encountered a conceptual question about transfer learning.
Suppose that I recall resnet50 model (pretrained module is True) and then, for my personal aim, I change the first layer (such that it accepts one channel image now) and the last layer (in this case, I add an additional fc layer to the end of the network). Now, my question is, what happens to the weights of the first layer (which I change that). In addition, what are the initial weights of my last layer (which wasn’t trained in the original one). Lastly, do the weights of 3 channel model fit properly to a one channel model?
You could try to add the three input channel contributions to get the same result as copying your input to all three channels. Ideally, inputs should be globally normalized to zero mean unit variance for that.
The weights of the first layer would be the same as the pretrained ResNet50 that you are using as you only changing the input size, assuming that you have not done any finetuning or training with target data.
Thank you for your reply. I have 2 more questions:
Are the number of weights for 1 channel input and 3 channel input the same? (in my opinion, they shouldn’t be the same and, therefore, how does the network pick up its weights for my 1 channel input?).
for the last part, what did you mean by ‘if learned weights of all three channels have similar distribution’.?
The number of weights for 1 channel input and 3 channel input won’t be same. In fact, number of channels in input layer always need to be matched in the first layer.
You may try changing the number of channels from 3 to 1 in the PyTorch CIFAR-10 tutorial. You can see the number of parameters with torch-summary package.
from torchsummary import summary
summary(net, ( 3, 32, 32))