I have to apply the following operations on a video (frame-wise)
- RGB to grayscale
- Center Crop
- Color Normalize
- Give the video (the sequence of the frames) as input to a model.
Moreover, I have to compute gradients over these operations later. I asked this Question and was suggested to use Kornia which supports gradients over these operations, which works well for me.
There is another problem though. The model expects an input of the shape (1, #frames, height, width) whereas the above operations expect the shape (#frames, #channels, height, width). So I need to apply
reshape() operation, due to which I am unable to compute the gradients. Kindly provide me with a workaround or point out what am I doing wrong.
The reshape operation is differentiable in pytorch. So it should not prevent you from computing any gradient.
Also I think you want to permute dimensions, not reshape them:
input = input.permute(1, 0, 2, 3).
Thanks a lot. Is the permute operation differentiable too?
Yes. In general, all the operations in pytorch that take continuous numbers and return continuous numbers are differentiable.