When get the intermediate features, it's get the OOM error

1

I want to get the intermediate features for calculating the perceptual loss, but get the OOM error, blow is my code:

class Vgg19(torch.nn.Module):
    def __init__(self):
        
        super(Vgg19, self).__init__()
        features = list(vgg19(pretrained=True).features)
        self.features = torch.nn.ModuleList(features).eval()
        
    def forward(self, x):
        
        results = []
        for ii, model in enumerate(self.features):
            x = model(x)
            
            if ii in {1, 6, 11, 20, 29}:
                results.append(x)
        return results

vgg = Vgg().to(torch.device("cuda"))
features_list = vgg(x)

when I get the intermediate features by above-mentioned code, I get the OOM error? why?
I train this model with 4 Nvidia v100-16G cards and use distributedataparallel mode.

2

You are storing 5 potentially large tensors and the memory requirement depends on all operations you are performing on this data or in your training script in general.
Could you describe your expectation, i.e. why is an OOM error not expected? Was the core working in another form, i.e. are you seeing the OOM only by storing these 5 tensors? If so, how are you processing them? Did you try to check the memory usage before the changes and afterwards?