Well apparently, to get an output with the same height and width as the input, padding is always needed when the conv2d kernel size is 2. But the pading is only needed in the begining or at the end, not both. I could pad on both sides and then remove the begining or the end afterwards. But is it gonna impact the runtime? Is there an efficient way to do padding on one side only?
F.pad is able to pad on each dimension separately and might be useful in your use case.