Rotate and rescale images in batch

I have an image named adv_patch which is a tensor.
I also have a batch of images with known bounding box locations, and a pretrained image detection network.
I want to apply this adv_patch to the batch of images, meaning i have to rescale adv_patch, rotate it, and put it on the image at each of the locations indicated by the bounding boxes.

The goal is to optimize adv_patch by minimizing the loss of the network w.r.t. adv_patch.
This loss would be defined by how well the network still detects the objects in the images from my batch after adv_patch being applied.

In the end i would like adv_patch to be able to reliably ‘fool’ my detector.

As I understand, this would require me to use operations over which autograd can compute gradiënts in order to be able to backpropagate all the way back to adv_patch. So the functions for rotating and rescaling in torchvision.transforms are not an option, as they perform only on PIL images and transforming would mean detaching from graph and no gradients

I had some success in using affine_grid and grid_sample to rotate and rescale a single adv_patch, but applying an adv_patch on multiple detections on a batch of images is a whole different case.

Is there any way I could make this work?

Where are you stuck currently?
I’ve written a simple rotation using a rotation matrix and meshgrid for batched inputs here.
Would that help somehow?

Thank you for trying to help me. I used .expand() to get a batch of adv_patch images, of the size [N, M, C, H, W] where N is the number of images in my batch I should apply the patch on, and M is the number of detections per image in my batch. The problem I’m facing right now is that i would like to rotate the images in my adv_patch batch, but each give them a different (random) rotation.
I also have to resize each of these images to a specific size, defined in another tensor.

Thanks for sharing your code for a batched rotation. I used the code of your answer in another thread to rotate a single tensor. The problem I’m facing is that I have to give a different rotation to each of the adv_patch images in my batch. I made a tensor the same size of my batch, containing random angles in [0, 2*pi]. How can i do the rotation taking as input the batch of adv_patch images and the batch of rotations and giving me a batch of adv_patch images, each one rotated along the angles in the rotation batch? Is it possible on a batch level or should i use a loop for that?

Thanks for your time, I figured it out! It came down to calculating a batch of rotation matrices theta and constructing a batch of affine_grid based on that.

The only problem is that the first call to affine_grid takes a long time. I suspect it’s doing some kind of initialization?

you can also use torchgeometry,
https://torchgeometry.readthedocs.io/en/latest/warp_affine.html

2 Likes

Thanks! That would have been very helpful! It will come in handy for future projects.