How to train FCN 8s network using 6 channeled spectral image dataset?

Im trying to train the FCN8s model using a multispectral image dataset (Hamlin state beach park High-resolution multispectral dataset - 6 channel data and 19 classes). I’m getting the image segmented (Pixel wise classification) when I trained the network with 3 channel as input. Now trying to train the net using all the 6 channels, For that, I have changed the input channels and the weight tensor size suitable for the 6 channel training with the same hyperparameters, but after training for the same number of epochs im getting all the pixels classified into 0 class.

Do you use pre-trained weights, or do you train from scratch?

In the initial layer, I have used random weights and for the rest of the layers I used pre-trained weights

The original first weight tensor is probably of size
[out_channels, in_channels, kernel_size, kernel_size] = [64, 3, 3, 3].

You could initialize your weight matrix using simply the original matrix repeated 2 times :

conv.weight = conv.weight.repeat([1, 2, 1, 1])

The new weight tensor will be of size [64, 6, 3, 3].

The only possible issue is, the original weights are trained with RGB images, whereas you will use it with IR images. But I don’t think it is a problem.