Very low train accuracy using ResNet and Efficientnet transfer learning medical image classification

vision
1

I implemented and tested DenseNet, ResNet18, ResNet50 and Efficientnet from pretrained models in pytorch torchvision. Only denseNet121 is working. The training and validation accuracy are both very low and does not improve. I also tried for resnet 18

for param in model_res.parameters():
    param.requires_grad = True

but didn’t work either. I have tried using augmentation, but since the starting acc was only 20% ish I don’t think that’s the major reason. I also used with or without normalization, or using SGD instead of Adam, change the learning rate. My dataset is a very balanced 4 category brain tumor medical images. What would be the problem?

My (partly) working code for DenseNet which has over 95% acc only after 5 epochs, for other models implementation I only replaced the model, and they all give very low acc from 20%

model = densenet121(weights=DenseNet121_Weights.DEFAULT).to(device)
num_ftrs = model.classifier.in_features #1024
model.classifier = nn.Linear(num_ftrs, 4)
model = model.to(device)

criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                 std=[0.229, 0.224, 0.225])

data_transforms = {
    'train':
    transforms.Compose([
        transforms.Resize((224,224)),
        # transforms.RandomHorizontalFlip(),
        transforms.AugMix(),
        transforms.ToTensor(),
        normalize
    ]),
    'validation':
    transforms.Compose([
        transforms.Resize((224,224)),
        transforms.ToTensor(),
        normalize
    ]),
}

losses = {'train':[], 'validation':[]}
accuracies = {'train':[], 'validation':[]}
num_epochs=5
def train_model(model, criterion, optimizer, num_epochs):
    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch+1, num_epochs))
        print('-' * 10)

        for phase in ['train', 'validation']:
            if phase == 'train':
                model.train()
            else:
                model.eval()

            running_loss = 0.0
            running_corrects = 0

            for inputs, labels in tqdm.tqdm(dataloaders[phase], desc=phase, leave=False):
                inputs = inputs.to(device)
                labels = labels.to(device)

                outputs = model(inputs)
                loss = criterion(outputs, labels)

                if phase == 'train':
                    optimizer.zero_grad()
                    loss.backward()
                    optimizer.step()

                _, preds = torch.max(outputs, 1)
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)

            epoch_loss = running_loss / len(image_datasets[phase])
            epoch_acc = running_corrects.double() / len(image_datasets[phase])
            losses[phase].append(epoch_loss)
            accuracies[phase].append(epoch_acc.item())

            print('{} loss: {:.4f}, acc: {:.4f}'.format(phase,
                                                        epoch_loss,
                                                        epoch_acc))
    return model


model_trained = train_model(model, criterion, optimizer, num_epochs=5)

Help me)