# Wrapping submodules in Sequential

**URL:** <https://discuss.pytorch.org/t/wrapping-submodules-in-sequential/151700>\
**Category:** vision\
**Created:** [May 15, 2022, 4:25am UTC](https://discuss.pytorch.org/t/wrapping-submodules-in-sequential/151700 "2022-05-15T04:25:14Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![seer\_mer](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/seer_mer/32/40173_2.png) [@seer\_mer](https://discuss.pytorch.org/u/seer_mer)\
**Post date:** [May 15, 2022, 4:25am UTC](https://discuss.pytorch.org/t/wrapping-submodules-in-sequential/151700/1 "2022-05-15T04:25:14Z")

</div>

Is the statement in the link below true?

> [@How to delete layer in pretrained model?](https://discuss.pytorch.org/t/how-to-delete-layer-in-pretrained-model/17648/27):
>
> InceptionV3 uses functional calls in its forward, which will be missing, if you wrap all submodules in an nn.Sequential container. In particular [this flatten operation](https://github.com/pytorch/vision/blob/dec8628d92c52e9b0d3f8816037870907b7d99f2/torchvision/models/inception.py#L174) will be missing. You could use forward hooks as described [here](https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6).

According to the response, functional calls will be missing when using nn.Sequential.  
However, I saw many repos use submodules with functional api in sequential, for example, the official convnext implementation:

> <https://github.com/facebookresearch/ConvNeXt/blob/main/models/convnext.py>

They used functional api (F.layer\_norm) inside submodule (class Block), and used nn.Sequential to wrap all the blocks.

---

<div class="post-metadata">

**Author:** ![Matias\_Vasquez](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/matias_vasquez/32/47847_2.png) [@Matias\_Vasquez](https://discuss.pytorch.org/u/Matias_Vasquez)\
**Post date:** [May 15, 2022, 3:51pm UTC](https://discuss.pytorch.org/t/wrapping-submodules-in-sequential/151700/2 "2022-05-15T15:51:48Z")

</div>

Maybe the easiest way to understand this is with a little example.

Here I have defined two Modules that perform functional stuff. The `SimpleBlock` multiplies by `2`. The `MainModel` uses this `SimpleBlock` in two different places and also performs some functional stuff (multiply by `3` and add `3`).

This can, of course, be something more complex like using layers, batch normalization, flatten or whatever you want to do.

```python
class SimpleBlock(torch.nn.Module):
    def __init__ (self):
        super(). __init__ ()

    def functional_stuff(self, x):
        return x * 2
    
    def forward(self, x):
        # Functional - Multiply by 2
        x = self.functional_stuff(x)
        return x

class MainModel(torch.nn.Module):
    def __init__ (self):
        super(). __init__ ()
        self.block_0 = SimpleBlock()
        self.block_1 = SimpleBlock()
        
    def functional_stuff(self, x):
        # This is inside a nn.Module
        x = self.block_0(x)

        # This is functional
        x = x * 3
        return x

    def forward(self, x):
        x = self.functional_stuff(x)

        # Functional - Add 3
        x = x + 3 

        # This is a nn.Module
        # The functional stuff indside this module is done
        # even if you extract the children with Sequential
        x = self.block_1(x)
        return x

```

We now use the `MainModel` and extract the children using `nn.Sequential`.

```python
input = torch.ones(2, 2)

model = MainModel()
seq_model = torch.nn.Sequential(*list(model.children()))

print(model(input))
print(seq_model(input))

```

If we now see the output generated, we can see that they do NOT yield the same result.

```python
# Output

# Model
tensor([[18., 18.],
        [18., 18.]])
# Sequential extraction
tensor([[4., 4.],
        [4., 4.]])

```

The full model gives the correct output.  
`([1 * 2] * 3 + 3) * 2 = ([2] * 3 + 3) * 2 = (6 + 3)*2 = (9)*2=18`

However the `nn.Sequential` did **not** multiply by `3` or added the `3`.  
`([1*2])*2 = 2*2 = 4`

This is because the functional stuff is wrapped inside this `nn.Module` called `SimpleBlock`. But the stuff on the `forward` method of our `MainModel` is not. We can even use `nn.Modules` that are in another function other than `forward` and call them from inside `forward`. But if they are **not** `nn.Modules` (or a container like `nn.Sqeuential` or others) they will not work.

In your example of `ConvNeXt`, this `F.layer_norm` is inside the class `LayerNorm` used there. This will work.

Also in the forward method of `ConvNeXt` we call another method. Inside this method. However, everything here is either a `nn.Module` or inside a container like `nn.Sequential` and `nn.ModuleList`.

So if we did something like this

```python
mod = torchvision.models.convnext_tiny(pretrained=True)
seq_mod = torch.nn.Sequential(*list(mod.children()))

mod.eval()
seq_mod.eval()

input = torch.rand(1, 3, 300, 300)

print(torch.all(mod(input) == seq_mod(input)))

```

We get `tensor(True)` as the output.  
If there was any other form of functional API on the forward of `ConvNeXt` (not the children but ONLY on `ConvNeXt`) then it would not work the same.

In the example given, there is a `flatten` function, which will break the pipeline if missing.

Hope this helps and makes it a bit clearer 😄