We had recently the same issue here. Do you see the same loading time when you use pretrained=False?
pretrained=False