Difference in Prediction scores of Pytorch and Keras model


I was trying to see if probability scores of pre-trained models from Keras and Pytorch match for the same images. However, I found that there is a slight difference.

I am confused, should this be the case?

The simple code is given below:
import torch
import torchvision.models as models
from PIL import Image
import numpy as np
import torch.nn.functional as F

mean = [0.485, 0.456, 0.406]
std = [0.229, 0.224, 0.225]

pytorchModel = models.resnet50(pretrained=True)

imagePath = ‘/home/jalwana/Data/Imagenet2012/temp/n02504013/training/n02504013_1830.JPEG’
image = np.asarray(Image.open(imagePath).resize((224, 224), Image.BILINEAR), dtype=np.float32)
image = image / 255.0
image = np.transpose(image, (2, 0, 1))

image[0, :, :] -= mean[0]
image[1, :, :] -= mean[1]
image[2, :, :] -= mean[2]
image[0, :, :] /= std[0]
image[1, :, :] /= std[1]
image[2, :, :] /= std[2]

image = torch.unsqueeze(torch.from_numpy(image), 0)
probs = F.softmax(pytorchModel(image), dim=1)

The keras model

from keras.applications.resnet50 import ResNet50
from keras.applications.resnet50 import preprocess_input

image = np.asarray(Image.open(imagePath).resize((224, 224), Image.BILINEAR), dtype=np.float32)
preprocessedImage = np.expand_dims(preprocess_input(image), axis=0)

model = ResNet50()
predictionsKeras = model.predict(preprocessedImage[:, :, ::-1])

print(probs[0, probs.argmax()])
print(predictionsKeras[0, predictionsKeras.argmax()])

For one image

pytorch resnet50 gave 0.8059

keras resnet50 gave 0.7875

the image belongs to ImageNet training dataset

Since prediction score is not a metric, should we be considerate of it?

Since both models were pretrained separately, I would expect some predictions to be a bit different.
Have you compared the validation performance of both models?
I’m not sure, how the ResNet was pretrained in Keras and what reference implementation and training was used.

1 Like