Getting different predictions each time I load model

vision
1

Hi all,

I’m using resnet50 with some added layers. When I go to run my predictions I get significantly different values for the same input image. When I train I’m using nn.DataParallel and when I’m trying to predict Im using the cpu. In sudo code, my evaluation is as such:

model = resnet_50_spin_pyramid_error()
model = model.to(torch.device("cpu"))
weights = "model/path.pth"
model.load_state_dict(
    torch.load(weights, map_location=torch.device("cpu"))["state_dict"],
    strict=False,
)
with torch.no_grad()
   model.eval()
   img = load(img_path)
   out = model(img)

Any ideas what might be going on? I am using nn.BatchNormalization but I think that should be turned off by being in eval mode.

Update: Using torch.manual_seed(your_seed_value) I’m able to get a consistent result, but I didn’t have a seed set during training. Is there any way to figure out the random state it was trained in?

Update: I wasn’t loading my weights correctly :slight_smile: fixing that it looks like it works.

Part of the model building function:

class SpinPyramidError(nn.Module):
    def __init__(self, *args, **kwargs) -> None:
        super().__init__(*args, **kwargs)
        self.dgcn_seg1 = spin(planes=32, ratio=2)
        self.dgcn_seg2 = spin(planes=32, ratio=2)
        self.conv2 = nn.Conv2d(32, 32, 3)
        self.fin_conv2 = nn.Conv2d(32, 2, 3)

    def forward(self, x):
        spin_256 = self.dgcn_seg1(x)
        _, c, h, w = spin_256.shape
        x2_128 = nn.MaxPool2d(kernel_size=2, stride=1)(x)
        spin_128 = self.dgcn_seg2(x2_128)
        spin_128_up = nn.functional.interpolate(spin_128, size=(h, w))
        x2 = torch.add(spin_128_up, spin_256)

        x3 = self.conv2(x)
        x4 = nn.ReLU(inplace=True)(x3)
        x5 = self.fin_conv2(x4)
        return x5

def resnet_50_spin_pyramid_error():
    model = deeplabv3_resnet50() # Load net
    model.classifier[4] = torch.nn.Conv2d(
        256, 32, kernel_size=(1, 1), stride=(1, 1)
    )  # Change final layer to 2 classes
    sp = SpinPyramidError()
    model.classifier.add_module("spin_pyramid", sp)
    return model