How to Implement a convolutional layer

Sure!
Here you can find a manual 2D conv implementation.
The 1D case should be straightforward by removing a spatial dimension:

batch_size = 2
channels = 5
h, w = 12, 12
image = torch.randn(batch_size, channels, h, w) # input image

kh, kw = 3, 3 # kernel size
dh, dw = 3, 3 # stride

# Create conv
conv = nn.Conv2d(5, 7, (kh, kw), stride=(dh, dw), bias=False)
filt = conv.weight

# Manual approach
patches = image.unfold(2, kh, dh).unfold(3, kw, dw)
print(patches.shape) # batch_size, channels, h_windows, w_windows, kh, kw

patches = patches.contiguous().view(batch_size, channels, -1, kh, kw)
print(patches.shape) # batch_size, channels, windows, kh, kw

nb_windows = patches.size(2)

# Now we have to shift the windows into the batch dimension.
# Maybe there is another way without .permute, but this should work
patches = patches.permute(0, 2, 1, 3, 4)
print(patches.shape) # batch_size, nb_windows, channels, kh, kw

# Calculate the conv operation manually
res = (patches.unsqueeze(2) * filt.unsqueeze(0).unsqueeze(1)).sum([3, 4, 5])
print(res.shape) # batch_size, output_pixels, out_channels
res = res.permute(0, 2, 1) # batch_size, out_channels, output_pixels
# assuming h = w
h = w = int(res.size(2)**0.5)
res = res.view(batch_size, -1, h, w)

# Module approach
out = conv(image)


print('max abs error ', (out - res).abs().max())
> max abs error  tensor(3.5763e-07, grad_fn=<MaxBackward1>)
3 Likes