Why is my CAM always predicting the same area?

vision
1

Here is the part of my code relevant to the issue:

    def forward_hook(module,input,output):
        activation.append(output)

    def backward_hook(module,grad_in,grad_out):
        grad.append(grad_out[0])


    model.layer4[-1].register_forward_hook(forward_hook)
    model.layer4[-1].register_backward_hook(backward_hook)
    grad=[]
    activation=[]


 loader_iter = iter(dataloader_test)
    for _ in range(50):
        data, target, meta = next(loader_iter)       
      
        for d, t, m in zip(data, target, meta):

            hm_dogs = []
            heatmap = []
            d, t = map(lambda x: x.to(device), (d, t))
            
            #remove batch size
            d = d.unsqueeze(0)
            output = model(d)

            output[:, 4].backward()
            #get the gradients and activations collected in the hook
            grads=grad[0].cpu().data.numpy().squeeze()
            fmap=activation[0].cpu().data.numpy().squeeze()

Basically, only the first sample is getting the right heatmap properly.

Edit:
I tried to iterate but basically the gradients and activations that are being hooked are all the same values for all samples.

Any ideas for me? Thank you in advance