Visualizing attention when using transfer learning method for images

Mona_Jalal · March 29, 2022, 10:04pm

I came across this paper but I wonder if I am using the PyTorch transfer learning tutorial, a code like

monajalal/Inception_V3-cats_and_dogs/blob/master/inception_v3_test.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n"
     ]
    }
   ],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]

This file has been truncated. show original

How can I apply this kinds of attention methodology to it since I don’t have access to the full network and I am using a model pretrained on ImageNet.

^ this is from this paper: Park, Jongchan, Sanghyun Woo, Joon-Young Lee, and In So Kweon. “A simple and light-weight attention module for convolutional neural networks.” International Journal of Computer Vision 128, no. 4 (2020): 783-798.

Also, Li, Xingjian, Haoyi Xiong, Zeyu Chen, Jun Huan, Ji Liu, Cheng-Zhong Xu, and Dejing Dou. “Knowledge Distillation with Attention for Deep Transfer Learning of Convolutional Networks.” ACM Transactions on Knowledge Discovery from Data (TKDD) 16, no. 3 (2021): 1-20.
is another example of what I am interested in for transfer learning using a pre-trained model.

I am open to any method such as GradCAM or RISE that could be applicable to visualizing the saliency/attention map in the transfer learning paradigm. If there are such existing notebooks for a starter you could point me to would be really great. (Fig 1.d)

thecho7 · March 30, 2022, 12:52am

You are finding how to draw a heatmap from the resulting attention.
I give you an useful notebook from GradCAM (pytorch-grad-cam/Class Activation Maps for Object Detection With Faster RCNN.ipynb at master · jacobgil/pytorch-grad-cam · GitHub)