Torchvision pretrained has accuracy gap to the reported value

Hi everyone, I’m trying to train something using the pretrained densenet201 from torchvision as base. First I tested the pretrained densenet201 to see how its actual accuracy (the reported accuracy is 77 - 77.5) is:

  1. By using densenet201 from torchvision.models, and follow the example code from the website:https://github.com/pytorch/examples/tree/42e5b996718797e45c46a25c55b031e6768f8440/imagenet
    I got the top-1 ImageNet accuracy of 76.896

  2. I also tried using the pretrained model from original densenet authors: https://github.com/liuzhuang13/DenseNet, convert the lua torch model into pytorch model, the test accuracy is still 76.896

Could someone help me clarify why there is a >0.3 accuracy drop of the pretrained model? Thanks

Hello, has anyone ever experienced the acc gap I mentioned for the pretrained model?

The original weights for those DenseNet models were trained in Torch with bicubic interpolation, so use bicubic interpolation in the Scale transform. The default in PyTorch is generally bilinear.

Also, accuracies of pretrained model don’t necessarily stay static over time. With changes in various parts of the system (cudnn, PyTorch mechanics, OpenCV/Pillow image processing, other libraries) you can see small changes over time, whether or not it’s statistically significant is another matter.

I benchmarked this model 1.5 years ago with whatever was the lastest versions of all dependencies and got > 77. Now it’s just under.

1.5 years ago w/ bicubic: 77.290 (22.710), 93.478 (6.522)
now w/ bicubic: 76.950 (23.050), 93.410 (6.590)
now w/ bilinear: 76.594 (23.406), 93.258 (6.742)

1 Like

Thank you for your valuable information, this is very helpful. It seems that the bicubic interpolation plays an important role. I also tested the ResNet pretrained models, they are perfectly matched with the paper’s results. And according to what you shown, even with bicuib interpolation, the current version of DenseNet is still under 77. Does it mean that there isn’t any place to find a better base that fits the original 77.2 result on Pytorch as of now?

Well, try it with bicubic on your system first, your numbers (that I presume are with bilinear scaling) are already higher than mine.

If you want to get that exact accuracy in test, hunt through past PyTorch versions and their associated libraries… I doubt it’s worth it for .34%. I think I did the 77.29 test around PyTorch 0.2.0 w/ CUDA 7.5 or 8 and CUDNN 6

Edit: Before you actually spend time doing that, just switch to dpn68b and be done with it :slight_smile: It’s a punchy little model w/ pretrained weights that’ll give you slightly higher accuracy than densenet-201… with 7.4m less parameters.

sure, I will try that. thanks!