When initialize a network, you need specify the corresponding input size in advance, such as
net = nn.Conv2d(3, 64)
net = nn.Linear(256, 64)
for Conv2d, you don’t need to specify the input image size except for channel depth. However, for Linear layer, you need to specify the input size, which means, the input size should be fixed.
Another problem is that, since Linear layer o = w*x+b, when your x dimension changes, you need change the dimension of weights. I don’t know whether you can deal with this.