Quick question, instead of random patterns if I want to divide the image into equal parts say, for example, 4 parts, and decide which parts are to be convolved. Would that boost the performance?
I’m not sure what the baseline would be.
If you are comparing a “full” conv vs. only applying the conv kernels to a subset, you could expect a speedup using the im2col approach.
Thank you so much for the quick reply. Yes, the baseline is full convolution. I would like to divide the image into n parts and apply convolution on only a few parts. Let us say, we have another two tensors. Difference = torch.randint(0,2,(1,1,128,128)) and value = torch.randint(0,2(1,1,128,128)). I would like to divide value into 16 equal parts and if all the values of difference tensor are zero in that part than that part of value will not be used in convolution. A sample code would be:
patches = padding(value).unfold(0,kernel_size+2,kernel_size).unfold(1,kernel_size+2,kernel_size).reshape(-1,1,kernel_size+2,kernel_size+2)
select = torch.not_equal(F.max_pool2d(difference, kernel_size=kernel_size+2, stride=kernel_size,padding=1),0).flatten()
value.index_put_((torch.tensor(0),d,i,j),F.conv2d(patches[select],weights,stride=1))
Now, my problem is I am not gaining any performance with this approach. In fact, this is slower than the full convolution. Could you please help me here? Also, it would be great if you could provide me any sample code how to use im2col for the approach you mention.
I’m not sure I understand the example, since it seems you are using unfold to create the patches but are then passing them to F.conv2d?
If you are unfolding the input, you would have to use a matmul to get the desired output as described in e.g. the nn.Unfold docs.
Patches are the 16 parts of the actual value tensor(128, 128). Each of the patches[select] will again be 3232. So, not all patches are used in convolution only the patches where select is true will be part of convolution. So, unfold here is creating a 16 3232 patches and weights are of size 9,1,3,3.