Hi all,
It seems like models in eval mode return different outputs depending on the batch size of the input provided.
It is my understanding this is not expected.
I already checked here, here (and in other 4 posts but the new user limit limits me to 2 links) but the answer is not obvious to me.
I created some dummy code to demonstrated the problem I’m having.
Any Idea if this is expected or why it is happening?
torch version: 1.10.1
python version: 3.9.7
import torch
import torchvision
model = torchvision.models.resnet18(pretrained=True, progress=True)
model.eval()
a = torch.ones((1, 3, 96, 96))
b = torch.ones((64, 3, 96, 96))
with torch.no_grad():
a_out = model(a)
b_out = model(b)
diff = torch.absolute(b_out[0] - a_out[0])
total_diff = torch.sum(diff)
average_diff = torch.mean(diff)
print(f"equal: {torch.equal(a_out[0], b_out[0])}")
print(f"rtol=0.00001: {torch.allclose(a_out[0], b_out[0])}")
print(f"rtol=0.0001: {torch.allclose(a_out[0], b_out[0], rtol=0.0001)}")
print(f"rtol=0.001: {torch.allclose(a_out[0], b_out[0], rtol=0.001)}")
print(f"rtol=0.01: {torch.allclose(a_out[0], b_out[0], rtol=0.01)}")
print(f"rtol=0.1: {torch.allclose(a_out[0], b_out[0], rtol=0.1)}")
print(f"total diff {total_diff}")
print(f"avg diff {average_diff}")
And this is the output:
equal: False
rtol=0.00001: False
rtol=0.0001: False
rtol=0.001: True
rtol=0.01: True
rtol=0.1: True
total diff 0.04874414950609207
avg diff 7.616273478561197e-07