Grad Cam error in final two layers of network

I am attempting Binary classification on lung cancer tumours. The network inputs are “masks” of the tumours, with everything outside of the mask being set to 0 as a background. I have been having several issues with the last layers of the network. For the first 3 layers, the grad cams are activating as expected, but for the final two they arent. The activations on the last layer are appearing nowhere near the actual tumours, but instead in the zero background as circles, or the layers arent activating at all. This is possibly reflected in the testing phase of the program, which always returns an accuracy near 50%, despite the validation accuracy being much higher consistently. For this I am using a Resnet architecture found here 3D-ResNets-PyTorch/resnet.py at master · kenshohara/3D-ResNets-PyTorch · GitHub but I have replaced the final fc layer with another conv layer and a global average pool.