Find similarity between featuremaps (rois)

Is there anyone knows how it is possible to find most relevant (most similar) parts between two 51277 tensor feature maps?

Actually in pytorch I defined two layer neural network that calculate similarity, however it seems it doesn’t work efficiency.

Any help please?