The meaning of 【return nn.Sequential (* layers) 】

1

I am a beginner. I have a simple question about Resnet-18.(but I really don’t understand) The following is the network structure code of Resnet-18.
I can’t understand the code,and I didn’t find a specific explanation in the reference books :
**return nn.Sequential (* layers)**
Does it mean that the last returned value is (layers),For example, if layers = [10,20], by executing this line of code, the final return value is still [10,20]. If this is the case, why not use return layers (I don’t know whether this is feasible)directly.

定义resnet18
'''
def conv3x3(in_planes, out_planes, stride=1):
    "3x3 convolution with padding"
    return nn.Conv2d(in_planes, out_planes, kernel_size=3, stride=stride,
                     padding=1, bias=False)

class BasicBlock(nn.Module):
    expansion = 1
    #inplanes其实就是channel,叫法不同
    def __init__(self, inplanes, planes, stride=1, downsample=None):
        super(BasicBlock, self).__init__()
        self.conv1 = conv3x3(inplanes, planes, stride)
        self.bn1 = nn.BatchNorm2d(planes)
        self.relu = nn.ReLU(inplace=True)
        self.conv2 = conv3x3(planes, planes)
        self.bn2 = nn.BatchNorm2d(planes)
        self.downsample = downsample
        self.stride = stride

    def forward(self, x):
        residual = x

        out = self.conv1(x)
        out = self.bn1(out)
        out = self.relu(out)

        out = self.conv2(out)
        out = self.bn2(out)
        #把shortcut那的channel的维度统一
        if self.downsample is not None:
            residual = self.downsample(x)

        out += residual
        out = self.relu(out)

        return out
    

class ResNet(nn.Module):
    def __init__(self, block, layers, num_classes=10):
        self.inplanes = 64
        super(ResNet, self).__init__()
        self.conv1 = nn.Conv2d(1, 64, kernel_size=7, stride=2, padding=3,   #因为mnist为（1，28，28）灰度图，因此输入通道数为1
                               bias=False)
        self.bn1 = nn.BatchNorm2d(64)
        self.relu = nn.ReLU(inplace=True)
        self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
        self.layer1 = self._make_layer(block, 64, layers[0])
        self.layer2 = self._make_layer(block, 128, layers[1], stride=2)
        self.layer3 = self._make_layer(block, 256, layers[2], stride=2)
        self.layer4 = self._make_layer(block, 512, layers[3], stride=2)
        self.avgpool = nn.AvgPool2d(7, stride=1)
        self.fc = nn.Linear(512 * block.expansion, num_classes)

        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
                m.weight.data.normal_(0, math.sqrt(2. / n))
            elif isinstance(m, nn.BatchNorm2d):
                m.weight.data.fill_(1)
                m.bias.data.zero_()

    def _make_layer(self, block, planes, blocks, stride=1):
        #downsample 主要用来处理H(x)=F(x)+x中F(x)和xchannel维度不匹配问题
        downsample = None
        #self.inplanes为上个box_block的输出channel,planes为当前box_block块的输入channel
        if stride != 1 or self.inplanes != planes * block.expansion:
            downsample = nn.Sequential(
                nn.Conv2d(self.inplanes, planes * block.expansion,
                          kernel_size=1, stride=stride, bias=False),
                nn.BatchNorm2d(planes * block.expansion),
            )

        layers = []
        layers.append(block(self.inplanes, planes, stride, downsample))
        self.inplanes = planes * block.expansion
        for i in range(1, blocks):
            layers.append(block(self.inplanes, planes))

        return nn.Sequential(*layers)

    def forward(self, x):
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.maxpool(x)

        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.layer4(x)

        x = self.avgpool(x)
        x = x.view(x.size(0), -1)
        x = self.fc(x)

        return x
    
def resnet18(pretrained=False, **kwargs):
    """Constructs a ResNet-18 model.
    Args:
        pretrained (bool): If True, returns a model pre-trained on ImageNet
    """
    #[2, 2, 2, 2]和结构图[]X2是对应的
    model = ResNet(BasicBlock, [2, 2, 2, 2], **kwargs)
    if pretrained: #加载模型权重
        model.load_state_dict(model_zoo.load_url(model_urls['resnet18']))
    return model

net  = resnet18()
`

``
3 Likes
2

layers in this snippet is a standard python list. First of all, python lists are not registered in a nn.Module which will lead to issues. That is way there exist a list-like layer which is a nn.Module.

So imagine layers consist of:
layers=[layer1,layer2,layer3]
and
seq_layers=nn.Sequential(*layers)

Secondly, nn.Sequential runs the three layers at once, this is, it takes the input, run layer1, take output1 and feed layer2 with it, take output2 and feed layer3 giving as result output3.
So this two blocks of code are equivalent:

x = ... # our input
for layer in layers:
   x = layer(x)

Which for the case would be:

x = ... # our input

x = layer1(x)
x = layer2(x)
x = layer3(x)

but using a nn.Sequential the same code can be called in a single line
x=seq_layers(x)

So nn.Sequential is a construction which is used when you want to run certain layers sequentially. It makes the forward to be readable and compact.

So in the code you are pointing to, they build different ResNet architectures with the same function. They return a nn.Sequential because they can run it without the functions knows “what’s inside”
It consist of 10 layers? it will run these 10, it’s a single one? no problem.

5 Likes
3

Thank you for your kind and detailed explanation. I’m worried the question is too simple to be answered.

Referring to your answer, I reread the code twice. I was fundamentally wrong before.

Now my understanding is: 【sequential】 is like a big box, in which 【layers】 are arranged according to the transmission order of neural networks. Then encapsulate the big box. When calling, only need to pass the actual parameters to 【sequential】, and the program will automatically execute layers according to the sequence. After executing the last layer, it will return the calculation result.

1 Like
4

Yes. That’s it. A last detail is this only works for single-input layers.
As you can see in the source code of its forward function:

    def forward(self, input):
        for module in self:
            input = module(input)
        return input

Aaand a minor correction:

in which 【layers】 are arranged according to the transmission order of neural networks

Layers are arranged according to the ordering of *layers. The method is not aware of which is transmission order

If you don’t set a proper ordering, an exception will raise

1 Like
5

thank you so much.It helped me a lot

6

good job ! Very good job and explantations.