RuntimeError: Trying to backward through the graph a second time. Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True

So I have gone through solutions already available on the forum. I’m using torchxrayvision and torchcam library. I needed Densenet121 pretrained weigths and torchcam to generate GradCAM for the model.

Minimal reproducible example I can give you is this

preds = model(rescaled_output.unsqueeze(0))
cam_extractors = [
    cam(class_idx=i, scores=preds) for i, _ in enumerate(model.pathologies)
]

rescaled_output just represents an image in Tensor of shape (1, 244, 244).
I’m anyways encountering the RuntimeError as following:

{
	"name": "RuntimeError",
	"message": "Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.",
	"stack": "---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Input In [24], in <cell line: 2>()
      1 preds = model(rescaled_output.unsqueeze(0))
----> 2 cam_extractors = [
      3     cam(class_idx=i, scores=preds) for i, _ in enumerate(model.pathologies)
      4 ]

Input In [24], in <listcomp>(.0)
      1 preds = model(rescaled_output.unsqueeze(0))
      2 cam_extractors = [
----> 3     cam(class_idx=i, scores=preds) for i, _ in enumerate(model.pathologies)
      4 ]

File /opt/homebrew/Caskroom/miniconda/base/envs/xray/lib/python3.10/site-packages/torchcam/methods/core.py:169, in _CAM.__call__(self, class_idx, scores, normalized, **kwargs)
    166 self._precheck(class_idx, scores)
    168 # Compute CAM
--> 169 return self.compute_cams(class_idx, scores, normalized, **kwargs)

File /opt/homebrew/Caskroom/miniconda/base/envs/xray/lib/python3.10/site-packages/torchcam/methods/core.py:193, in _CAM.compute_cams(self, class_idx, scores, normalized, **kwargs)
    178 \"\"\"Compute the CAM for a specific output class.
    179 
    180 Args:
   (...)
    190         the k-th element of the input batch for class index equal to the k-th element of `class_idx`.
    191 \"\"\"
    192 # Get map weight & unsqueeze it
--> 193 weights = self._get_weights(class_idx, scores, **kwargs)
    195 cams: List[Tensor] = []
    197 with torch.no_grad():

File /opt/homebrew/Caskroom/miniconda/base/envs/xray/lib/python3.10/site-packages/torchcam/methods/gradient.py:100, in GradCAM._get_weights(self, class_idx, scores, **kwargs)
     98 \"\"\"Computes the weight coefficients of the hooked activation maps.\"\"\"
     99 # Backpropagate
--> 100 self._backprop(scores, class_idx, **kwargs)
    102 self.hook_g: List[Tensor]  # type: ignore[assignment]
    103 # Global average pool the gradients over spatial dimensions

File /opt/homebrew/Caskroom/miniconda/base/envs/xray/lib/python3.10/site-packages/torchcam/methods/gradient.py:59, in _GradCAM._backprop(self, scores, class_idx, retain_graph)
     57     loss = scores.gather(1, torch.tensor(class_idx, device=scores.device).view(-1, 1)).sum()
     58 self.model.zero_grad()
---> 59 loss.backward(retain_graph=retain_graph)

File /opt/homebrew/Caskroom/miniconda/base/envs/xray/lib/python3.10/site-packages/torch/_tensor.py:492, in Tensor.backward(self, gradient, retain_graph, create_graph, inputs)
    482 if has_torch_function_unary(self):
    483     return handle_torch_function(
    484         Tensor.backward,
    485         (self,),
   (...)
    490         inputs=inputs,
    491     )
--> 492 torch.autograd.backward(
    493     self, gradient, retain_graph, create_graph, inputs=inputs
    494 )

File /opt/homebrew/Caskroom/miniconda/base/envs/xray/lib/python3.10/site-packages/torch/autograd/__init__.py:251, in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    246     retain_graph = create_graph
    248 # The reason we repeat the same comment below is that
    249 # some Python versions print out the first line of a multi-line function
    250 # calls in the traceback and some print out the last line
--> 251 Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
    252     tensors,
    253     grad_tensors_,
    254     retain_graph,
    255     create_graph,
    256     inputs,
    257     allow_unreachable=True,
    258     accumulate_grad=True,
    259 )

RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward."
}

Thank you in advance for your help.

Are you using retain_graph=True? If so, could you explain why this is needed?

I’m not using retain_graph=True. So I’m using GradCAM to get activation heatmaps of the neurons. The GradCAM uses backpropagtion values to generate them as far as I know hence that’s why the graph is being accessed second time. Since I’m using torchxrayvision directly and not creating a custom model or training script hence I don’t know where I can change the attribute of retain_graph.

import torchxrayvision

model = xrv.models.DenseNet(weights="densenet121-res224-all")
model.eval()

That’s all I’m doing to get the model.

EDIT: Checked the source code. while calling self._backprop it remains False as default value.