DepthWise1dConv with causal padding. Need help

1

Explanation

I want to implement a DepthWise1dConv with causal padding. That is, padding should be applied before the signal starts.

I followed this wonderful resource to understand depth-wise convolutions, although explained for the 2D case, I assume it its directly applicable to the 1D case.

From the torch.nn.Conv1d documentation, whenever padding is added, it is added to both sides of the input (and it can be observed in the module variable _reversed_padding_repeated_twice. But this is not the desired behavior.

However, whenever I try to set the padding manually by using torch.nn.functional.pad I get an error. What am I missing here?

Minimal reproducible example

import torch

x = torch.rand(1, 64, 384) # (B, C, L)
conv1d = torch.nn.Conv1d(in_channels=64, out_channels=64, kernel_size=17, padding='same', groups=64)
print(f"{conv1d(x).shape=}, {conv1d._reversed_padding_repeated_twice=}")

p3d = (0, 0, 16, 0, 0, 0)
x = torch.nn.functional.pad(x, p3d)
print(f"{x.shape=}")
conv1d = torch.nn.Conv1d(in_channels=80, out_channels=64, kernel_size=17, padding='valid', groups=80)
conv1d(x)

Output

conv1d(x).shape=torch.Size([1, 64, 384]), conv1d._reversed_padding_repeated_twice=[8, 8]
x.shape=torch.Size([1, 80, 384])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[98], line 10
      8 x = torch.nn.functional.pad(x, p3d)
      9 print(f"{x.shape=}")
---> 10 conv1d = torch.nn.Conv1d(in_channels=80, out_channels=64, kernel_size=17, padding='valid', groups=80)
     11 conv1d(x)

File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/conv.py:300, in Conv1d.__init__(self, in_channels, out_channels, kernel_size, stride, padding, dilation, groups, bias, padding_mode, device, dtype)
    298 padding_ = padding if isinstance(padding, str) else _single(padding)
    299 dilation_ = _single(dilation)
--> 300 super().__init__(
    301     in_channels, out_channels, kernel_size_, stride_, padding_, dilation_,
    302     False, _single(0), groups, bias, padding_mode, **factory_kwargs)

File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/conv.py:92, in _ConvNd.__init__(self, in_channels, out_channels, kernel_size, stride, padding, dilation, transposed, output_padding, groups, bias, padding_mode, device, dtype)
     90     raise ValueError('in_channels must be divisible by groups')
     91 if out_channels % groups != 0:
---> 92     raise ValueError('out_channels must be divisible by groups')
     93 valid_padding_strings = {'same', 'valid'}
     94 if isinstance(padding, str):

ValueError: out_channels must be divisible by groups

In tensorflow

import tensorflow as tf

x = tf.random.uniform((1, 64, 384))
print(f"{x.shape=}")
causal_pad = tf.keras.layers.ZeroPadding1D((16,0))(x)
print(f"{causal_pad.shape=}")
dw_conv = tf.keras.layers.DepthwiseConv1D(
    kernel_size=17,
    strides=1,
    dilation_rate=1,
    padding='valid',
    use_bias=True,
    depthwise_initializer='glorot_uniform')(causal_pad)
print(f"{dw_conv.shape=}")

Output

x.shape=TensorShape([1, 64, 384])
causal_pad.shape=TensorShape([1, 80, 384])
dw_conv.shape=TensorShape([1, 64, 384])
2

I’m not deeply familiar with Keras, but based on the shape I assume you are padding the temporal dimension.
In this case this code should work:

import torch

x = torch.rand(1, 64, 384) # (B, C, L)
conv1d = torch.nn.Conv1d(in_channels=64, out_channels=64, kernel_size=17, padding='same', groups=64)
print(f"{conv1d(x).shape=}, {conv1d._reversed_padding_repeated_twice=}")

p3d = (16, 0, 0, 0, 0, 0)
x = torch.nn.functional.pad(x, p3d)
print(f"{x.shape=}")
conv1d = torch.nn.Conv1d(in_channels=64, out_channels=64, kernel_size=17, padding='valid', groups=64)
out = conv1d(x)
print(out.shape)
# torch.Size([1, 64, 384])