Can this be achieved?

1

Hi There!

Can something like this list of Conv1d layers be achieved using a single Conv2d or Conv2dTransposed? I tried thinking really hard but was unable to think of something clever!

assert in_channels < out_channels
assert features.shape == (batch_size, in_channels, signal_length)

convs = nn.ModuleList(
    [nn.Conv1d(in_channels=in_channels, out_channels=out_channels, kernel_size=1)
     for _ in range(num_layers)])

# [batch_size, in_channels, signal_length] →
# list of [batch_size, out_channels, signal_length]
features = [conv(features) for conv in convs]

assert len(features) == num_layers

# list of [batch_size, out_channels, signal_length] →
# [batch_size, num_layers, out_channels, signal_length]
features = torch.stack(features, dim=1)

Thanks!

2

It seems that you are trying to stack the layers in convs on top of each other.
Since the in_channels and out_channels are fixed to x and y for each layer, this won’t work. if x!=y.

Could you explain a bit, what you mean by # [batch_size, num_layers, y, signal_length]?
Should your output have this shape? If so, what is the difference between num_layers and y?
In your code you define the number of out_channels as y. So num_layers should be y.

3

Hi There!

I updated the example code to explain more about the shape!

4

Thanks for the updated code!
I think this should even be possible with a Conv1d using out_channels=out_channels*num_layers and a view on the result tensor.

I created a small example which demonstrates this approach.
For an easy comparison of both methods I set bias=False and initialized the weigths for each “layer” with numbers in range(num_layers). So the first layer uses all zeros, the second ones, the third twos, etc.

in_channels = 3
num_layers = 12
out_channels = 6
batch_size = 10
signal_length = 5
kernel_size = 1
features = torch.randn(batch_size, in_channels, signal_length)
features2 = features.clone()

convs = nn.ModuleList(
    [nn.Conv1d(in_channels=in_channels, out_channels=out_channels, kernel_size=kernel_size, bias=False)
     for _ in range(num_layers)])
for i, conv in enumerate(convs):
    conv.weight.data.fill_(1 * i)

features = [conv(features) for conv in convs]
features = torch.stack(features, dim=1)
print('First layer: ', features[:, 0])
print('Second layer: ', features[:, 1])

# Second approach
conv1d = nn.Conv1d(
    in_channels=in_channels,
    out_channels=out_channels*num_layers,
    kernel_size=kernel_size,
    bias=False)

# Set manually weight to same as used in convs ModuleList
weight = [torch.ones(out_channels, in_channels, kernel_size) * i for i in range(num_layers)]
weight = torch.cat(weight)
conv1d.weight.data = weight

features2 = conv1d(features2)
features2 = features2.view(batch_size, num_layers, out_channels, -1)
print('First layer: ', features2[:, 0])
print('First layer: ', features2[:, 1])

# Print error for all layers
for idx in range(num_layers):
    print('Error in layer {}: {}'.format(
        idx, torch.abs(features[:, idx] - features2[:, idx]).sum()))
5

Did not think to use the view method like that, thank you!