Why does ResNet50 training on down sampled TinyImageNet not converge?

1

Hi all, I am trying to train a ResNet50 for scratch on down sampled TinyImageNet dataset. Unfortunately my network dose not converge on test data. I use the following transformer and optimizer to train the model. Any advice would be really appreciated.

transforms = {
    "train": transforms.Compose([
        transforms.Resize(32),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406],std=[0.229, 0.224, 0.225])]),
       "test": transforms.Compose([
        transforms.Resize(32),
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406],std=[0.229, 0.224, 0.225])]),}

trainset= datasets.ImageFolder(os.path.join(data_dir, "train"), transforms["train"])
trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size,shuffle=True, num_workers=0)

testset= datasets.ImageFolder(os.path.join(data_dir, "test"), transforms["test"])
testloader = torch.utils.data.DataLoader(testset, batch_size=batch_size,shuffle=False, num_workers=0)

optimizer = optim.SGD(net.parameters(), lr=0.01, momentum=0.9, weight_decay=1e-2) 
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=10)