Rotate and rescale images in batch

vision
1

I have an image named adv_patch which is a tensor.
I also have a batch of images with known bounding box locations, and a pretrained image detection network.
I want to apply this adv_patch to the batch of images, meaning i have to rescale adv_patch, rotate it, and put it on the image at each of the locations indicated by the bounding boxes.

The goal is to optimize adv_patch by minimizing the loss of the network w.r.t. adv_patch.
This loss would be defined by how well the network still detects the objects in the images from my batch after adv_patch being applied.

In the end i would like adv_patch to be able to reliably ‘fool’ my detector.

As I understand, this would require me to use operations over which autograd can compute gradiënts in order to be able to backpropagate all the way back to adv_patch. So the functions for rotating and rescaling in torchvision.transforms are not an option, as they perform only on PIL images and transforming would mean detaching from graph and no gradients

I had some success in using affine_grid and grid_sample to rotate and rescale a single adv_patch, but applying an adv_patch on multiple detections on a batch of images is a whole different case.

Is there any way I could make this work?

1 Like
2

Where are you stuck currently?
I’ve written a simple rotation using a rotation matrix and meshgrid for batched inputs here.
Would that help somehow?

3

Thank you for trying to help me. I used .expand() to get a batch of adv_patch images, of the size [N, M, C, H, W] where N is the number of images in my batch I should apply the patch on, and M is the number of detections per image in my batch. The problem I’m facing right now is that i would like to rotate the images in my adv_patch batch, but each give them a different (random) rotation.
I also have to resize each of these images to a specific size, defined in another tensor.

Thanks for sharing your code for a batched rotation. I used the code of your answer in another thread to rotate a single tensor. The problem I’m facing is that I have to give a different rotation to each of the adv_patch images in my batch. I made a tensor the same size of my batch, containing random angles in [0, 2*pi]. How can i do the rotation taking as input the batch of adv_patch images and the batch of rotations and giving me a batch of adv_patch images, each one rotated along the angles in the rotation batch? Is it possible on a batch level or should i use a loop for that?

4

Thanks for your time, I figured it out! It came down to calculating a batch of rotation matrices theta and constructing a batch of affine_grid based on that.

The only problem is that the first call to affine_grid takes a long time. I suspect it’s doing some kind of initialization?

5

you can also use torchgeometry,
https://torchgeometry.readthedocs.io/en/latest/warp_affine.html

3 Likes
6

Thanks! That would have been very helpful! It will come in handy for future projects.

7

quick update: this can be found in kornia:
https://kornia.readthedocs.io/en/latest/geometry.transform.html#kornia.geometry.transform.rotate
AND
https://kornia.readthedocs.io/en/latest/geometry.transform.html#kornia.geometry.transform.scale

8

Hey Simenthys
I have a similar problem as you posted here. I want to apply a rotated patch to an image and pass it through a neural net and calculate the loss. Then do backprop and revise the patch.

So, do you have any update on the best approach that one can use? Which of the answers do you suggest?

9

Thank you a lot @ptrblck for your code. I created the rotation function I wanted to have with your lines 21-33.
I have just changed the centor location to x_mid=im.size(2) / 2 and y_mid=im.size(2)/ 2. as the +1 is already included into the im.size. Hope I am right. It improved quite a lot my rotation.