I have two end-to-end networks. The input of the network is an image with size N by N. The output of the first network is an N by N mask that mask[i,j]=k means that pixel k in the input should change location to i-th row and j-th column of the image (so the pixels of the input image will be switched with each other). Then this new image will be input to the second network and at the end, there is a loss function. Is there any way to have this relocation for pixels such that the backpropagation algorithm works well?
Thanks