After extracting activation map from intermediate layers how can I pass it to the submodules (skip connection blocks) of another network?

vision
1

I have extracted activation maps from unet (https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix) and this is discussed here (https://discuss.pytorch.org/t/how-to-extract-intermediate-feature-maps-from-u-net/27084). Now I want to pass the values to UnetSkipConnectionBlock module.

I was unable to pass it by this manner
out_image = self.model(input, activation=self.activation, hook_idx=self.hook_idx).

Many thanks in advance.

2

Hi,

I checked the two links and the implementation of the UnetSkipConnectionBlock. Unfortunately I cannot really tell what your are trying to do and what your self.model actually implements. Could you elaborate more and share some code?

3

Thank you for your response…

I want to pass the ‘activation’ and ‘hook_idx’ variables to each UnetSkipConnectBlock so that it can be accessed at each layer and modified. The parameters are common for every (UnetSkipConnectBlock) layer. I think if I make down, up and submodule (in UnetSkipConnectBlock) global and change it in forward pass (of UnetSkipConnectBlock) it will work…? Currently trying something similar…

4

If you register the hooks to the other network, would’t that do the job? You would be able to access the activation dictionary (which is global) and index it by setting the name you want when you register the hook.

activation = {}
def get_activation(name):
    def hook(model, input, output):
        activation[name] = output.detach()
    return hook

model = UnetGenerator(1, 1, 5)
model.unet_block_1.register_forward_hook(get_activation('block1'))
model.unet_block_2.register_forward_hook(get_activation('block2'))
model.unet_block_3.register_forward_hook(get_activation('block3'))

model2 = UnetGenerator(1, 1, 5)
model2.unet_block_1.register_forward_hook(get_activation('block1'))
model2.unet_block_2.register_forward_hook(get_activation('block2'))
model2.unet_block_3.register_forward_hook(get_activation('block3'))

Since the blocks of the 2 models have the same name, they will index the same activation in the dictionary. Then, if you want to modify it in a way that is not overriding, you can just adapt the hook logic accordingly.

5

That makes sense! Thank you for the reply!