Custom Convolution Dot Product

22

(Update: Thankyou very much i did solved my issue, i replaced it with nested loops to get what i wanted)

23

hey, @ lucamocerino did you find a solution to your problem? I am too trying to modify the dot multiplication in the convolution layer. Thanks

24

Avoid nested loops! Try to have as much of a functional code as you can. Loops don’t parallelize well on the GPU.

I also don’t know for sure what that @ operator is.

From the documentation, it seems to be
https://pytorch.org/docs/stable/torch.html#torch.bmm

On his example, the shapes are not both 3 dimensions, and they just match the size in one of the dimensions, 16. So, I am assuming that multiplication can possible be broadcasted, thus https://pytorch.org/docs/stable/torch.html#torch.matmul. However I could get an actual confirmation to which operation is that @ mapping to.

1 Like
25

@ is matrix multiplication in python3. Yes you are right about the nested loops part. I am trying to do all of it with functional codes.

27

I was just able to make it work for me with the code bellow.
PS.: When pasting here I decided to put the hard code values on the top, I don’t think I messed up anything.

Let me know if it doesn’t run still. I am put a Gist here for you.

batch_size = 32
size_img = 32 
size_k = 5
padd_k = 2
c_in, c_out = 64, 64

batch_sample = np.random.normal(size=(batch_size, c_in, size_img, size_img))
kernel = np.random.normal(size=(c_out, c_in, size_k, size_k))

device = torch.device('cuda:0') \
  if torch.cuda.is_available() else torch.device('cpu')
print(device)

_batch_sample = torch.from_numpy(batch_sample).float().cuda()
_kernel = torch.from_numpy(kernel).float().cuda()

def conv2d_spatial(x, filters):
  return torch.nn.functional.conv2d(x, filters, padding=padd_k

def conv2d_spatial2(x, filters):
  unfold = torch.nn.functional.unfold(x, size_k, padding=padd_k) 
  kernels_flat = filters.view(c_out, -1)
  out = kernels_flat @ unfold
  return out.view(-1, c_out, size_img, size_img)
1 Like
28

Thanks alot for your code, i really appreciate it. Actually my problem was slightly different, i had to simulate behavior of certain adders and multiplier inside “@” part [1]. I did solved it exactly like your code using numpy for my multipliers and adders.

30

@eduardo4jesus I believe the size of the output of the convolution is not the image size, I think it’s (height_image - height_filter + 2 * padding) + 1

1 Like
31

Thanks for pointing that out, at some point I fixed that up. I was just revisiting the topic looking for whether or not is the same implementation and mathematical equations for convolution with stride > 1.
Though I have used the code below, I have an assert somewhere else to guarantee stride == 1.

def _conv2d(cls, input, weight, stride=1, padding=0, dilation=1):
    # print('Custom __conv2d')
    # From https://cs231n.github.io/convolutional-networks/
    # O = (W - F + 2P)/S + 1, constrained so that O is integer.
    # Since we only support stride == 1
    # O = (W - F + 2P) + 1
    out_size = input.size(-2) - weight.size(-2) + 2*padding[0] + 1

    input_unfolded = F.unfold(input, weight.size()[2:], dilation, padding, stride)
    weight_flat = weight.reshape(weight.size(0), -1)
    conv = weight_flat@input_unfolded; del weight_flat; del input_unfolded;
    output = conv.reshape(input.size(0), weight.size(0), out_size, out_size)
    return output

Would the back-propagation for convolution with stride be the same as regular convolution? How the derivative for stride > 1 would be?