Expecting a face similarity loss function

Hi gurus

I am finding a face similarity loss function which accepts two tensors as input, any idea?

My goal is to find a loss function which can do backward, as SSIM is very good for replacement of L2, but it detects the global structure, I want to compare the face only.

The expected usage

loss = 1 - face_similarity(tensor1, tensor2)
loss.backward()

I know there is a facenet-pytorch project, MTCNN + ResNet(vggface2) combo which can compare the faces, but it breaks the gradients.

Any clue will be appreciated!