Initializing custom first layer (with increased channel) of pretrained ResNet model

1

I have images from satellite data that has 6 channels. Usual RGB channels and 3 other channels. Now, I am trying to fine tune a pretrained resnet model. I understand how to change the first convolution layer to accommodate for 3 additional input channel (from this discussion post but I have other concern. So, in the new custom first layer, I want to set the kernel parameters for RGB channel to be same as the pretrained resnet model. But for the new 3 channel, i would like to initialize the kernel parameters as mean of (at corresponding coordinates). kernels of pretrained resnet RGB channels.

Not sure how I can do that. Any suggestion/example would be super helpful!

1 Like
2

I would create a new conv layer with 6 channels and modify its parameters by hand:

trained_kernel = trained_layer.weight # this is the pretrained RGB kernel
new_conv = nn.ConvLayer2d( 6, out_channels, kernel_size, ...)
new_conv.weight[:,:3] = trained_kernel
# if d is the dimension over which you want to average:
new_conv.weight[:,3:] = torch.mean(trained_kernel, d).unsqueeze(d).expand_as(trained_kernel)
2 Likes