How to save inner nn.Sequential layer's output

vision
1

VGG(
(features): Sequential(
(0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(0_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
** (1): ReLU(inplace=True)**
** (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))**
** (2_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**
** (3): ReLU(inplace=True)**
** (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)**
** (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))**
** (5_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**
** (6): ReLU(inplace=True)**
** (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))**
** (7_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**
** (8): ReLU(inplace=True)**
** (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)**
** (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))**
** (10_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**
** (11): ReLU(inplace=True)**
** (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))**
** (12_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**
** (13): ReLU(inplace=True)**
** (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))**
** (14_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**
** (15): ReLU(inplace=True)**
** (16): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)**
** (17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))**
** (17_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**
** (18): ReLU(inplace=True)**
** (19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))**
** (19_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**
** (20): ReLU(inplace=True)**
** (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))**
** (21_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**
** (22): ReLU(inplace=True)**
** (23): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)**
** (24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))**
** (24_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**
** (25): ReLU(inplace=True)**
** (26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))**
** (26_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**
** (27): ReLU(inplace=True)**
** (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))**
** (28_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**
** (29): ReLU(inplace=True)**
** (30): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)**
** )**
** (classifier): Sequential(**
** (0): Linear(in_features=25088, out_features=4096, bias=True)**
** (0_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**
** (1): ReLU(inplace=True)**
** (2): Dropout(p=0.5, inplace=False)**
** (3): Linear(in_features=4096, out_features=4096, bias=True)**
** (3_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**
** (4): ReLU(inplace=True)**
** (5): Dropout(p=0.5, inplace=False)**
** (6): Linear(in_features=4096, out_features=1000, bias=True)**
** (6_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**
** )**
)

  • My model’s shape is like this.

  • I saw that hook is not work inner nn.seq module

  • and also when I tried that thing, the ofmap of feature.0 layer and ifmap of feature.0_linear_quant is different

  • Then, If I want conv2d or 0_linear_quant layer’s output feature map, what can I do?

import torch.nn as nn
import torch.utils.model_zoo as model_zoo
import math

all = [
** ‘VGG’, ‘vgg11’, ‘vgg11_bn’, ‘vgg13’, ‘vgg13_bn’, ‘vgg16’, ‘vgg16_bn’,**
** ‘vgg19_bn’, ‘vgg19’,**
]

model_urls = {
** ‘vgg11’: ‘https://download.pytorch.org/models/vgg11-bbd30ac9.pth’,**
** ‘vgg13’: ‘https://download.pytorch.org/models/vgg13-c768596a.pth’,**
** ‘vgg16’: ‘https://download.pytorch.org/models/vgg16-397923af.pth’,**
** ‘vgg19’: ‘https://download.pytorch.org/models/vgg19-dcbb9e9d.pth’,**
}

class VGG(nn.Module):

** def init(self, features, num_classes=1000):**
** super(VGG, self).init()**
** self.features = features**
** self.classifier = nn.Sequential(**
** nn.Linear(512 * 7 * 7, 4096),**
** nn.ReLU(inplace=True),**
** nn.Dropout(),**
** nn.Linear(4096, 4096),**
** nn.ReLU(inplace=True),**
** nn.Dropout(),**
** nn.Linear(4096, num_classes),**
** )**
** self._initialize_weights()**

** def forward(self, x):**
** x = self.features(x)**
** x = x.view(x.size(0), -1)**
** x = self.classifier(x)**
** return x**

** def initialize_weights(self):**
** for m in self.modules():**
** if isinstance(m, nn.Conv2d):**
** n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels**
** m.weight.data.normal(0, math.sqrt(2. / n))**
** if m.bias is not None:**
** m.bias.data.zero_()**
** elif isinstance(m, nn.BatchNorm2d):**
** m.weight.data.fill_(1)**
** m.bias.data.zero_()**
** elif isinstance(m, nn.Linear):**
** n = m.weight.size(1)**
** m.weight.data.normal_(0, 0.01)**
** m.bias.data.zero_()**

def make_layers(cfg, batch_norm=False):
** layers = []**
** in_channels = 3**
** for v in cfg:**
** if v == ‘M’:**
** layers += [nn.MaxPool2d(kernel_size=2, stride=2)]**
** else:**
** conv2d = nn.Conv2d(in_channels, v, kernel_size=3, padding=1)**
** if batch_norm:**
** layers += [conv2d, nn.BatchNorm2d(v), nn.ReLU(inplace=True)]**
** else:**
** layers += [conv2d, nn.ReLU(inplace=True)]**
** in_channels = v**
** return nn.Sequential(layers)*

cfg = {
** ‘A’: [64, ‘M’, 128, ‘M’, 256, 256, ‘M’, 512, 512, ‘M’, 512, 512, ‘M’],**
** ‘B’: [64, 64, ‘M’, 128, 128, ‘M’, 256, 256, ‘M’, 512, 512, ‘M’, 512, 512, ‘M’],**
** ‘D’: [64, 64, ‘M’, 128, 128, ‘M’, 256, 256, 256, ‘M’, 512, 512, 512, ‘M’, 512, 512, 512, ‘M’],**
** ‘E’: [64, 64, ‘M’, 128, 128, ‘M’, 256, 256, 256, 256, ‘M’, 512, 512, 512, 512, ‘M’, 512, 512, 512, 512, ‘M’],**
}

def vgg11(pretrained=False, model_root=None, kwargs):
** “”“VGG 11-layer model (configuration “A”)”""
** model = VGG(make_layers(cfg[‘A’]), kwargs)
** if pretrained:**
** model.load_state_dict(model_zoo.load_url(model_urls[‘vgg11’], model_root))**
** return model**

def vgg11_bn(kwargs):
** “”“VGG 11-layer model (configuration “A”) with batch normalization”""
** kwargs.pop(‘model_root’, None)**
** return VGG(make_layers(cfg[‘A’], batch_norm=True), kwargs)

def vgg13(pretrained=False, model_root=None, kwargs):
** “”“VGG 13-layer model (configuration “B”)”""
** model = VGG(make_layers(cfg[‘B’]), kwargs)
** if pretrained:**
** model.load_state_dict(model_zoo.load_url(model_urls[‘vgg13’], model_root))**
** return model**

def vgg13_bn(kwargs):
** “”“VGG 13-layer model (configuration “B”) with batch normalization”""
** kwargs.pop(‘model_root’, None)**
** return VGG(make_layers(cfg[‘B’], batch_norm=True), kwargs)

def vgg16(pretrained=False, model_root=None, kwargs):
** “”“VGG 16-layer model (configuration “D”)”""
** model = VGG(make_layers(cfg[‘D’]), kwargs)
** if pretrained:**
** model.load_state_dict(model_zoo.load_url(model_urls[‘vgg16’], model_root))**
** return model**

def vgg16_bn(kwargs):
** “”“VGG 16-layer model (configuration “D”) with batch normalization”""
** kwargs.pop(‘model_root’, None)**
** return VGG(make_layers(cfg[‘D’], batch_norm=True), kwargs)

def vgg19(pretrained=False, model_root=None, kwargs):
** “”“VGG 19-layer model (configuration “E”)”""
** model = VGG(make_layers(cfg[‘E’]), kwargs)
** if pretrained:**
** model.load_state_dict(model_zoo.load_url(model_urls[‘vgg19’], model_root))**
** return model**

def vgg19_bn(kwargs):
** “”“VGG 19-layer model (configuration ‘E’) with batch normalization”""
** kwargs.pop(‘model_root’, None)**
** return VGG(make_layers(cfg[‘E’], batch_norm=True), kwargs)

  • base model is like this and

def duplicate_model_with_quant(model, bits, overflow_rate=0.0, counter=10, type=‘linear’):
** “”“assume that original model has at least a nn.Sequential”""**
** assert type in [‘linear’, ‘minmax’, ‘log’, ‘tanh’]**
** if isinstance(model, nn.Sequential):**
** l = OrderedDict()**
** for k, v in model._modules.items():**
** if isinstance(v, (nn.Conv2d, nn.Linear, nn.BatchNorm1d, nn.BatchNorm2d, nn.AvgPool2d)):**
** l[k] = v**
** if type == ‘linear’:**
** quant_layer = LinearQuant(’{}_quant’.format(k), bits=bits, overflow_rate=overflow_rate, counter=counter)**
** elif type == ‘log’:**
** # quant_layer = LogQuant(’{}_quant’.format(k), bits=bits, overflow_rate=overflow_rate, counter=counter)**
** quant_layer = NormalQuant(’{}_quant’.format(k), bits=bits, quant_func=log_minmax_quantize)**
** elif type == ‘minmax’:**
** quant_layer = NormalQuant(’{}_quant’.format(k), bits=bits, quant_func=min_max_quantize)**
** else:**
** quant_layer = NormalQuant(’{}quant’.format(k), bits=bits, quant_func=tanh_quantize)**
** l[’{}{}_quant’.format(k, type)] = quant_layer**
** else:**
** l[k] = duplicate_model_with_quant(v, bits, overflow_rate, counter, type)**
** m = nn.Sequential(l)**
** return m**
** else:**
** for k, v in model._modules.items():**
** model._modules[k] = duplicate_model_with_quant(v, bits, overflow_rate, counter, type)**
** return model**

  • this is how LinearQuant layer generated
2

You can use forward hooks to store intermediate activations as shown in this example.

PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier. :wink:

3 
    activation = {}
    ofmap = {}
    def get_ofmap(name):
        def hook(model, input, output):
            ofmap[name] = output.detach()
        return hook
    def get_activation(name):
        def hook(model, input, output):
            activation[name] = input[0].detach()
        return hook

    for idx, (data, target) in enumerate(tqdm.tqdm(ds, total=n_sample)):
        n_passed += len(data)
        data =  Variable(torch.FloatTensor(data)).cuda()
        indx_target = torch.LongTensor(target)
        h1 = model.module.model._modules["features"]._modules.get("0_linear_quant").register_forward_hook(get_activation('0_linear_quant'))
        h2 = model.module.model._modules["features"]._modules.get("0").register_forward_hook(get_ofmap('fea0'))

        output = model(data)
        bs = output.size(0)

        torch.save(activation['0_linear_quant'], './LQ0_IFMAP.pt')
        torch.save(ofmap['fea0'], './CONV0_OFMAP.pt')

Thanks for your help!

But I tried that solution before but the output of h1 and h2 is different.
Because 0_LinearQuant layer’s input feature map is output of features.0 layer, h1 and h2 should be same.

4

It seems to work using this small example:

act_in = {}
act_out = {}
def get_hook(name):
    def hook(m, input, output):
        act_in[name] = input[0].detach()
        act_out[name] = output.detach()
    return hook
        

model = nn.Sequential(
    nn.Conv2d(3, 6, 3, 1, 1),
    nn.ReLU(),
    nn.Flatten(),
    nn.Linear(6*24*24, 10),
    nn.ReLU(),
    nn.Linear(10, 10)
)

for name, module in model.named_children():
    module.register_forward_hook(get_hook(name))

x = torch.randn(1, 3, 24, 24)
out = model(x)

keys = list(act_in.keys())
for key_out, key_in in zip(keys[:-1], keys[1:]):
    print('Comparing output of {} to input of {}'.format(key_out, key_in))
    print((act_out[key_out] == act_in[key_in]).all())

> Comparing output of 0 to input of 1
tensor(True)
Comparing output of 1 to input of 2
tensor(True)
Comparing output of 2 to input of 3
tensor(True)
Comparing output of 3 to input of 4
tensor(True)
Comparing output of 4 to input of 5
tensor(True)

Feel free to post an executable code snippet to reproduce this issue.

5

So I modify that code for the model that I use

VGG(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (0_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (1): ReLU(inplace=True)
    (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (2_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (3): ReLU(inplace=True)
    (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (5_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (6): ReLU(inplace=True)
    (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (7_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (8): ReLU(inplace=True)
    (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (10_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (11): ReLU(inplace=True)
    (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (12_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (13): ReLU(inplace=True)
    (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (14_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (15): ReLU(inplace=True)
    (16): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (17_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (18): ReLU(inplace=True)
    (19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (19_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (20): ReLU(inplace=True)
    (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (21_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (22): ReLU(inplace=True)
    (23): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (24_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (25): ReLU(inplace=True)
    (26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (26_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (27): ReLU(inplace=True)
    (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (28_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (29): ReLU(inplace=True)
    (30): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (classifier): Sequential(
    (0): Linear(in_features=25088, out_features=4096, bias=True)
    (0_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (1): ReLU(inplace=True)
    (2): Dropout(p=0.5, inplace=False)
    (3): Linear(in_features=4096, out_features=4096, bias=True)
    (3_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (4): ReLU(inplace=True)
    (5): Dropout(p=0.5, inplace=False)
    (6): Linear(in_features=4096, out_features=1000, bias=True)
    (6_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
  )
)

This is the model that I use right now and this is from the GitHub below.

and right now, when I change the code as this

    for idx, (data, target) in enumerate(tqdm.tqdm(ds, total=n_sample)):
        n_passed += len(data)
        data =  Variable(torch.FloatTensor(data)).cuda()
        indx_target = torch.LongTensor(target)
        for name, module in model.module.named_children():
            for name, module in module.named_children():
                for name, module in module.named_children():
                    module.register_forward_hook(get_hook(name))
        #h1 = model.module.model._modules["features"]._modules.get("0_linear_quant").register_forward_hook(get_activation('0_linear_quant'))
        #h2 = model.module.model._modules["features"]._modules.get("0").register_forward_hook(get_ofmap('fea0'))

        output = model(data)
        bs = output.size(0)

        keys = list(act_in.keys())
        for key_out, key_in in zip(keys[:-1], keys[1:]):
            print('Comparing output of {} to input of {}'.format(key_out, key_in))
            print((act_out[key_out] == act_in[key_in]).all())

Error is like this

Comparing output of 0 to input of 0_linear_quant
tensor(True, device='cuda:0')
Comparing output of 0_linear_quant to input of 1
tensor(True, device='cuda:0')
Comparing output of 1 to input of 2
tensor(True, device='cuda:0')
Comparing output of 2 to input of 2_linear_quant
Traceback (most recent call last):
  File "/home/sunyoung/research/quantization/quant/bin/quantize", line 33, in <module>
    sys.exit(load_entry_point('pytorch-playground', 'console_scripts', 'quantize')())
  File "/home/sunyoung/research/quantization/pytorch-playground/quantize.py", line 107, in main
    misc.eval_model(model_raw, val_ds_tmp, ngpu=1, n_sample=args.n_sample, is_imagenet=is_imagenet)
  File "/home/sunyoung/research/quantization/pytorch-playground/utee/misc.py", line 230, in eval_model
    print((act_out[key_out] == act_in[key_in]).all())
RuntimeError: The size of tensor a (4096) must match the size of tensor b (224) at non-singleton dimension 3

I don’t know why this happen because when I execute inference code without that, it doesn’t have problem.

6

Forward hooks can be registered to modules and if you are using calls from the functional API in the forward method, e.g. as seen here, the intermediate activation will be manipulated.
In the liked case the activation will be flattened, which would yield the shape mismatch error.

1 Like
8

I found a problem, it is because of the type I access

        for name, module in model.module.named_children():
            for name, module in module.named_children():
                for name, module in module.named_children():
                    module.register_forward_hook(get_hook(name))

this part. How to I access to each layer by for loop?

model.module.model._modules["features"]._modules.get("0_linear_quant").register_forward_hook(get_hook('0_linear_quant'))

I access to the tensor like this before… I couldn’t access with named_children.

9

Your loop might work, if you have nested modules, where the inner modules are the built-in layers.
If the model definition is more complicated (e.g. using mixed custom modules and layers), you might need to add conditions via if isinstance(module, nn.Conv2d) etc.

1 Like