<< extracting stats from activated layers - training >>

2
  1. You would have to call register_forward_hook on the particular layer, not the complete model/module. Based on your code it looks like you are working with a DataParallel model, since you are accessing the .module attribute. If that’s the case, you would need to address each layer separately:
net.module.features[2].register_forward_hook(save_activation('features.2'))
net.module.features[6].register_forward_hook(save_activation('features.6'))
net.module.features[10].register_forward_hook(save_activation('features.10'))
net.module.features[13].register_forward_hook(save_activation('features.13'))

The name passed to save_activation is just for your own convenience. :wink:

  1. You could just register the hook for the last epoch, pass some flag to save_activations if you are currently dealing with the last epoch, or just overwrite the activations using:
def save_activation(name):
    def hook(model, input, output):
        activations[name] = [(epoch,
        np.float(output.min().detach().data), 
        np.float(output.max().detach().data), 
        np.float(output.mean().detach().data), 
        np.float(output.std().detach().data))]
    return hook
  1. It depends on your use case etc. I personally like to use dicts as you can make sure which activation you are currently working with. You could use whatever data structure fits your needs, but I’m quite paranoid about using plain lists as I cannot be sure to look at the right layer (e.g. what if the list was somehow extended). :wink:

If you want to visualize some stats of your model using Tensorboard, have a look at this Ignite example, where some handlers are used to do this.

1 Like