Hi,
I would like to modify the coordinates of points used in the kernel during the convolution operation for an entire CNN. I don’t want to use a regular grid kernel but to adapt the shape of it depending on its location on the “image”.
Would you have tips on how to do that in Pytorch?
Can I simply add functions in my Python main and keep the Pytorch engine intact or do I need to modify the backend in order to keep the GPU/CUDA performances?
Thank you,
CO
I would recommend to not focus on the performance and backend at the beginning as your use case is not trivial.
Would the grid be used in fixed windows or would the window size also be changed depending on the image location?
In the first step, I would recommend to use either a nested loop and make sure your method is working as expected and then use unfold to create the windows and calculate the output via a matrix multiplication, if that fits your use case.
Thanks for your answer.
The grid size will stay the same, I just want to modify the coordinates of the feature points used.
If it helps for understanding, I did it in Caffe by modifying the im2col operation. Instead of taking a regular grid to compute the output of the function, I have just modified the position of the points depending on the location of the kernel center in the feature maps.
And if I have understood correctly the documentation, the unfold pytorch function does exactly the same operation. So it looks like the function I need to modify, isn’it?
So do you think it’s possible to modify the im2col pytorch operation, rebuild pytorch framework and directly use it for already built networks?
That could be possible, but I would rather create a new method (via a custom C++ extension) than manipulating native functions.
This tutorial shows how to implement such a method.