I have an image named
adv_patch which is a tensor.
I also have a batch of images with known bounding box locations, and a pretrained image detection network.
I want to apply this
adv_patch to the batch of images, meaning i have to rescale
adv_patch, rotate it, and put it on the image at each of the locations indicated by the bounding boxes.
The goal is to optimize
adv_patch by minimizing the loss of the network w.r.t.
This loss would be defined by how well the network still detects the objects in the images from my batch after
adv_patch being applied.
In the end i would like
adv_patch to be able to reliably ‘fool’ my detector.
As I understand, this would require me to use operations over which autograd can compute gradiënts in order to be able to backpropagate all the way back to
adv_patch. So the functions for rotating and rescaling in torchvision.transforms are not an option, as they perform only on PIL images and transforming would mean detaching from graph and no gradients
I had some success in using
grid_sample to rotate and rescale a single
adv_patch, but applying an
adv_patch on multiple detections on a batch of images is a whole different case.
Is there any way I could make this work?
Where are you stuck currently?
I’ve written a simple rotation using a rotation matrix and
meshgrid for batched inputs here.
Would that help somehow?
Thank you for trying to help me. I used
.expand() to get a batch of
adv_patch images, of the size [N, M, C, H, W] where N is the number of images in my batch I should apply the patch on, and M is the number of detections per image in my batch. The problem I’m facing right now is that i would like to rotate the images in my
adv_patch batch, but each give them a different (random) rotation.
I also have to resize each of these images to a specific size, defined in another tensor.
Thanks for sharing your code for a batched rotation. I used the code of your answer in another thread to rotate a single tensor. The problem I’m facing is that I have to give a different rotation to each of the
adv_patch images in my batch. I made a tensor the same size of my batch, containing random angles in [0, 2*pi]. How can i do the rotation taking as input the batch of
adv_patch images and the batch of rotations and giving me a batch of
adv_patch images, each one rotated along the angles in the rotation batch? Is it possible on a batch level or should i use a loop for that?
Thanks for your time, I figured it out! It came down to calculating a batch of rotation matrices
theta and constructing a batch of
affine_grid based on that.
The only problem is that the first call to
affine_grid takes a long time. I suspect it’s doing some kind of initialization?
Thanks! That would have been very helpful! It will come in handy for future projects.
I have a similar problem as you posted here. I want to apply a rotated patch to an image and pass it through a neural net and calculate the loss. Then do backprop and revise the patch.
So, do you have any update on the best approach that one can use? Which of the answers do you suggest?
Thank you a lot @ptrblck for your code. I created the rotation function I wanted to have with your lines 21-33.
I have just changed the centor location to x_mid=im.size(2) / 2 and y_mid=im.size(2)/ 2. as the +1 is already included into the im.size. Hope I am right. It improved quite a lot my rotation.