Classification accuracy changes when using registered forward hook

vision
1

Dear

I use pytorch 2.2.0 and cuda 12.1.
In my code, I have assigned a forward hook on each Relu layer of Resnet18 pretrained on Imagenet (from PyTorch). During evaluation mode, I modify activation maps outputing from Relu layer, and measuring the model classification accuracy.
The issue is that every time I run the code, the classification accuracy on validation set of ImageNet is different. Actually the number of truly classified samples are different in each run.

part of the code I modify the activation maps:

def modf_activations(self, module, input, output):
output = (self.weight[self.layer_counter].view(1, -1, 1, 1).cuda() * output).type(torch.float)
self.layer_counter +=1
return output

I thought this issue could be because of the pytorch version. The first version I had was pytorch 1.13. Once I run my code on pytorch 2.2 and cuda 12.1 the problem is still there.

Even the time I do not modify activation maps and I just return unchanged activation maps like below, the number of truly samples between different runs are different. This difference is not one or two sample, it is between 100 to 200.

def modf_activations(self, module, input, output):
return output

In another test, I do not assign any forward hook on the Resnet18. In this situation, every time I run the code, the classification accuracy is the same. So, in conclusion, in the presence of forward hook, the classification accuracy is not stable.

Based on these inputs, I would be appreciate if you could tell me what other things I can try to find the reason for such different output .

thanks.

2

Could you post a minimal and executable code snippet reproducing the issue, please?

3

Dear @ptrblck

please find below the minimal code. Please set the variable “add_To_be_Set” which is the address to a dataset directory (in my case validation set of ImageNet).

While I was preparing the minimal code, I noticed the part raise the bug. If you look at the below code, every time I run it, it gives the same classification accuracy.


import torchvision.models as models
import torch
import numpy as np

from torchvision import models, datasets
from torch.utils.data import DataLoader


class modifying_activationmaps:
    def __init__(self):
        self.current_activationmaps = []
        self.length = []
        self.weight = []
        self.layer_counter = 0
    def collect_dim_per_relu(self,module, input, output):
        self.length.append(output.shape[1])

    def modf_activations(self, module, input, output):
        #output = (self.weight[self.layer_counter].view(1, -1, 1, 1).cuda() * output).type(torch.float)
        #self.layer_counter +=1
        return output


def register_hook_modf_act(module):
    name = module.__class__.__name__
    if name.find('ReLU') != -1:
        module.register_forward_hook(acts_obj.modf_activations)

acts_obj = modifying_activationmaps()
model_cnn = models.resnet18(weights=models.ResNet18_Weights.IMAGENET1K_V1)
transformation = models.ResNet18_Weights.IMAGENET1K_V1.transforms()
model_cnn.apply(register_hook_modf_act)

model_cnn.eval()
model_cnn.cuda()

add_To_be_Set = 'ADRESS TO VALIDATIONSET OF IMAGENET'

dataset_dir = datasets.ImageFolder(add_To_be_Set, transformation)
data_loader = torch.utils.data.DataLoader(dataset_dir, batch_size=100, shuffle=False)


true_classified_samples = 0
for inputs, labels in data_loader:
    outputs = model_cnn(inputs.cuda())
    pred = torch.argmax(outputs,dim=1).detach().cpu()
    true_classified_samples += len(torch.where(pred==labels)[0])
    acts_obj.layer_counter = 0
print( 'true classified ',true_classified_samples, ' accuracy:',true_classified_samples/50000)

The other part of my code is the part where I add two forward hooks. The first one, is for collecting the depth of output of each Relu layer. Then, I remove the registered hooks. Finally, I add another hook to modify activations (in this version I do not modify). So, when I have these two together, I get different accuracies among different runs. However, I don’t know why it is such a case and how to handle it.

import torchvision.models as models
import torch
import numpy as np

from torchvision import models, datasets
from torch.utils.data import DataLoader


class modifying_activationmaps:
    def __init__(self):
        self.current_activationmaps = []
        self.length = []
        self.weight = []
        self.layer_counter = 0
    def collect_dim_per_relu(self,module, input, output):
        self.length.append(output.shape[1])

    def modf_activations(self, module, input, output):
        #output = (self.weight[self.layer_counter].view(1, -1, 1, 1).cuda() * output).type(torch.float)
        #self.layer_counter +=1
        return output

collect_relu_length_hook = []
def register_hook_collect_length(module):
    name = module.__class__.__name__
    if name.find('ReLU') != -1:
        collect_relu_length_hook.append(module.register_forward_hook(acts_obj.collect_dim_per_relu))

def register_hook_modf_act(module):
    name = module.__class__.__name__
    if name.find('ReLU') != -1:
        module.register_forward_hook(acts_obj.modf_activations)

acts_obj = modifying_activationmaps()
model_cnn = models.resnet18(weights=models.ResNet18_Weights.IMAGENET1K_V1)
transformation = models.ResNet18_Weights.IMAGENET1K_V1.transforms()
model_cnn.apply(register_hook_collect_length)
fake_input = torch.rand((1,3,224,224))
model_cnn(fake_input)
for hook in collect_relu_length_hook:
    hook.remove()
model_cnn.apply(register_hook_modf_act)

model_cnn.eval()
model_cnn.cuda()

add_To_be_Set = 'ADRESS TO VALIDATIONSET OF IMAGENET'

dataset_dir = datasets.ImageFolder(add_To_be_Set, transformation)
data_loader = torch.utils.data.DataLoader(dataset_dir, batch_size=100, shuffle=False)


true_classified_samples = 0
for inputs, labels in data_loader:
    outputs = model_cnn(inputs.cuda())
    pred = torch.argmax(outputs,dim=1).detach().cpu()
    true_classified_samples += len(torch.where(pred==labels)[0])
    acts_obj.layer_counter = 0
print( 'true classified ',true_classified_samples, ' accuracy:',true_classified_samples/50000)