Eval function seems to not turn every layer to eval mode

1

Hello I am trying to train a CNN network. During training it trains fine but when I go to testing I get that error : “File “/.local/lib/python3.10/site-packages/torch/nn/functional.py”, line 2560, in group_norm
_verify_batch_size([input.size(0) * input.size(1) // num_groups, num_groups] + list(input.size()[2:]))
File “/lib/python3.10/site-packages/torch/nn/functional.py”, line 2448, in _verify_batch_size
raise ValueError(f"Expected more than 1 value per channel when training, got input size {size}”)" .

I understand is because the normalization layer expects a batch bigger than 1. I read online that using .eval() function should make the normalization layer accept a batch of one. I am doing that but I still get the same error. Bellow is how I implement the layers of my model.

class ConvEncoder(nn.Module):
    def __init__(self, in_features, encoder_h=256, enc_width=(3, 2, 2, 2, 2, 2),
                 dropout=0., projection_head=False, enc_downsample=(3, 2, 2, 2, 2, 2),max_training_length=90):
        super().__init__()
        self.encoder = nn.Sequential()

        for i, (width, downsample) in enumerate(zip(enc_width, enc_downsample)):
            if i == 0:

                self.encoder.add_module("Encoder_{}".format(i), nn.Sequential(
                    nn.Conv1d(in_features, encoder_h, width, padding=width // 2),
                    nn.MaxPool1d(downsample),
                    nn.Dropout2d(dropout),
                    nn.GroupNorm(encoder_h , encoder_h),
                    nn.GELU(),
                ))
            else:

                self.encoder.add_module("Encoder_{}".format(i), nn.Sequential(
                    nn.Conv1d(in_features, encoder_h, width, padding=width // 2),
                    nn.MaxPool1d(downsample),
                    nn.Dropout2d(dropout),
                    nn.GroupNorm(encoder_h , encoder_h),
                    nn.GELU(),
                ))
            in_features = encoder_h

        if projection_head:
            self.encoder.add_module("projection-1", nn.Sequential(
                nn.Conv1d(in_features, in_features, 1),
                nn.Dropout2d(dropout*2),
                nn.GroupNorm(in_features, in_features),
                nn.GELU()
            ))
2

I don’t think this is the case for nn.GroupNorm since according to the docs:

This layer uses statistics computed from input data in both training and evaluation modes.

which means that enough values are needed to compute the stats. If you cannot increase the batch size you could try to lower the num_groups values.
Your code is unfortunately incomplete and not executable so I don’t know how the model is created and what the input is.