How to set the right transformations for inference

vision
1

Hi,

I have trained a classification network with the following transformations:

data_transforms = {
        'train': transforms.Compose([
            transforms.RandomResizedCrop(size=224, scale=(0.5, 1.0)),
            transforms.RandomAffine(15),
            transforms.ColorJitter(brightness=0.1, contrast=0.1, saturation=0.1, hue=0.1),
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ]),
        'val': transforms.Compose([
            transforms.Resize(256),
            transforms.CenterCrop(224),
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ]),
    }

Training accuracy is 99.99% and validation one is 99.65%, which is very good. Let’s name the above transformations train_transforms1 and val_transforms1, respectively.

Now if I change the validation transformations to the following, called val_transforms2:

'val': transforms.Compose([
            transforms.Resize((224,224)),
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ]),

then the validation accuracy drops significantly to 97.78%.

Now again if I change the training transformations to the following, called train_transforms2:

'train': transforms.Compose([
            transforms.Resize((224,224)),
            transforms.RandomAffine(15),
            transforms.ColorJitter(brightness=0.1, contrast=0.1, saturation=0.1, hue=0.1),
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ]),

then the results are inverse: very high accuracy with val_transforms2 and not so high accuracy with val_transforms1!

So I guess if in training we do RandomResizedCrop() then we should do Resize() and CenterCrop() at test time?

Is there a general rule for setting test time transformations according to training time ones? How should I best set test time transformations according to the train_transforms1 and train_transforms2 above?

Thank you very much for your suggestions!

2 Likes
2

Yes! The test data and training data should come from the same “domain”, if that’s possible.
If you train and test with different scales of your images, you will most likely get worse results.

In general, you should provide the same scales of the images, since the conv kernels learn to extract features based on a certain resolution.

1 Like
3

@ptrblck Thanks. That certainly makes sense, but did not totally answer my questions though.

Have look at Pytorch official tutorial on transfer learning: https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html, which have the following transformations:

data_transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

Now in the validation transforms, if you replace the two lines transforms.Resize(256),transforms.CenterCrop(224) by simply transforms.Resize((224,224)) then the accuracy drops! Why resizing to 256 and then center-crop 224 better than directly resizing to 224?

2 Likes
4

I see the issue here.
Also RandomResizedCrop scales the image to scale=(0.08, 1.0) by default, which seems to contradict my claim.

One possible explanation would be, that the model with the best validation accuracy is saved and reused later in the tutorial (code).
so in other words, you use the best model applying the ‘val’ transformation.

Do you see the same behavior, if you remove the CenterCrop, just resize the image, and train the model again?