Pytorch Tutorial for Neural Style Transfer

1

The above tutorial uses a pre-trained neural VGG network but does not adjust the images for mean or standard deviation. Since the Pytorch model zoo was trained using these transforms I would have thought that would reduce the accuracy of the model. I guess since its style transfer it might not matter but I was puzzled as to the reason this wasn’t done?

Any comments / suggestions

Thanks

John

2

Well the reason is simple: to make it simpler :wink:

Actually you are right, VGG has been trained with preprocessed images, so it may be cleaner to do the same on the input images for neural transfert. I simply noticed that it was not improving the results while adding some lines of code in the tutorial.

If you want to try with preprocess correction you can do:

# pre and post processing for images
img_size = 512 
prep = transforms.Compose([transforms.Scale(img_size),
                           transforms.ToTensor(),
                           transforms.Lambda(lambda x: x[torch.LongTensor([2,1,0])]), #turn to BGR
                           transforms.Normalize(mean=[0.40760392, 0.45795686, 0.48501961], #subtract imagenet mean
                                                std=[1,1,1]),
                           transforms.Lambda(lambda x: x.mul_(255)),
                          ])
postpa = transforms.Compose([transforms.Lambda(lambda x: x.mul_(1./255)),
                           transforms.Normalize(mean=[-0.40760392, -0.45795686, -0.48501961], #add imagenet mean
                                                std=[1,1,1]),
                           transforms.Lambda(lambda x: x[torch.LongTensor([2,1,0])]), #turn to RGB
                           ])
postpb = transforms.Compose([transforms.ToPILImage()])
def postp(tensor): # to clip results in the range [0,1]
    t = postpa(tensor)
    t[t>1] = 1    
    t[t<0] = 0
    img = postpb(t)
    return img

(source: https://github.com/leongatys/PytorchNeuralStyleTransfer/blob/master/NeuralStyleTransfer.ipynb)

3

Thanks very much, thought that was likely to be the case but just wanted to check. I notice that you have set the std dev to zero wheras in the pytorch model zoo documentation is is set to std = [0.229, 0.224, 0.225]. I guess that using this will make the clipping problem worse such that the target image will then lose significant information.

4

@fromLittleAcorns

Leon Gatys is using his VGG19 Conv model that was converted from Caffe to PyTorch. So the pre-processing is going to be different for that model, than the model zoo models which were trained in PyTorch. So the code that @alexis-jacq linked to is not correct for the model which his style transfer tutorial uses.

1 Like
5

Thanks, I’m getting some issues when I use preprocessing of the sort recommended for the Pytorch model zoo, will explore further tomorrow as it’s late in UK now

6

Oh yes, I forgot that. Thanks @ProGamerGov