To help interpret the neural network, calculation of saliency map is often used. A saliency map is calculated by using backpropagation to compute the gradients of logits (of the correct label) with respect to the input of the network (image, time-series…).
The calculated saliency map is a tensor of same shape as input and includes both positive and negative values. In related work, people always use the absolute value of saliency map to determine the important regions of the input.
Why is that so? To my understanding, the positive values in the saliency map are the regions of the input that push the prediction towards the correct class, while the negative values push the prediction to the other-class. Is my understanding correct? Therefore, to extract the regions that include information that is typical for the correct class, we should only take the regions of the positive saliency map. Right?