I have built a resnet 50 for classification task .I saved the model after training is completed . However when I load back the model and test the accuracy using the test set . It can give me a overall result 89% accuracy . But different accuracy can be obtained if I retest the test set .(test set is the always the same ).is it a problem about the model . I assume the accuracy should be always the same for same input .And the accuracy is actually fluctuating from 89 to 89.5 .(not a big difference) .is this expected ? I am pretty sure my coding and algorithm is absolutely right .
You might get non-deterministic outputs based on some operations such as non-deterministic cudnn kernels as explained in the reproducibility docs.
If you follow these docs, you should get deterministic outputs.