However, I think it is not right to use transforms.RandomResizedCrop(size=(128, 128)) for testing my network, because it randomly crops the image. If I want to compare my results to those from the official leaderboard, how do I have to configure test_transform?
For example, if a test image is 10000x10000 pixels, taking the 128x128 center crop before resizing to say (160x160 or 192x192) might result in a huge mismatch between the size of objects your model is used to seeing at training time and what it is now being tested on. Resizing to a known size before taking the center crop is a way to help alleviate this problem.
Overall, this issue is somewhat tricky to deal with even with resizing (see Touvron et. al for more details).