Equivalent tensorflow loss in pytorch

tensorflow have the loss function “tf.nn.sparse_softmax_cross_entropy_with_logits”. I saw an implementation of log function like this

 def log_loss(logits,labels,name)
    SE_loss = tf.nn.sparse_softmax_cross_entropy_with_logits
    return tf.reduce_sum(SE_loss(logits=logits, labels=label), [1, 2], name=name)

What would be the equivalent loss in pytorch and how to use this loss to have exact same answer as of the log_loss function provided given same logits and labels tensors

Here is how i am testing it but the pytorch cross entropy loss is not working

input = torch.randn((20, 30, 30,2), requires_grad=True)
target = torch.empty((20,30,30), dtype=torch.long).random_(1)
output = loss(input, target)
logits= input.detach().numpy()
labels= target.detach().numpy()

graph = tf.Graph()
with tf.Session(graph=graph) as sess:
    output_tf= tf.nn.sparse_softmax_cross_entropy_with_logits(labels=tf.convert_to_tensor(labels, dtype=tf.int32), logits=tf.convert_to_tensor(logits, dtype=tf.float32))
    output_tf= tf.reduce_sum(output_tf, [1, 2], name="tfloss")

    print("*"*50)
    print("tensorflow: ",sess.run(output_tf))
    print(output)
    print("*"*50)#

I get this error

RuntimeError: Assertion `input0 == target0 && input2 == target1 && input3 == target2’ failed. size mismatch (got input: 20x30x30x2, target: 20x30x30) at /pytorch/aten/src/THNN/generic/SpatialClassNLLCriterion.c:61

When i run simply tensorflow part, i get following type of results.


tensorflow: [782.8956 814.34094 819.82416 814.04565 804.5945 817.7538 816.1295
776.78687 785.20825 801.3092 802.24805 802.3621 846.7362 792.4359
768.00555 820.16675 782.90857 846.25085 793.9464 799.1505 ]


Length of the output is 20

I want to manipulate my input and output for pytorch cross entropy such that results of tensorflow code and pytorch cross entropy code come out as same. Looking forward to hearing from you. Thanks in advance for putting thought into it.

The class dimension should be in dim1 in your model output, so this code should work:

criterion = nn.CrossEntropyLoss()
input = torch.randn((20, 2, 30, 30), requires_grad=True)
target = torch.empty((20,30,30), dtype=torch.long).random_(2)
output = criterion(input, target)

I’m not sure, if the TF code returns an unreduced loss, but if so you could set reduction='none' in your criterion instantiation.

it resolves the issue but the answer is not same as tensorflow. even the dimensions are not same as of tensorflow code. Not sure how cross entropy is working in terms of dimensions in pytorch. Code is given below.

import torch.utils.data as Data
import torchvision
from torch.autograd import Variable
from torchvision import transforms
import config as cfg
import os
from torch.utils.data import Dataset, DataLoader
import scipy.io as sio
from PIL import Image
import h5py
import numpy as np
import math
import cv2
from torchvision import transforms
from PIL import Image
import h5py
from tensorboardX import SummaryWriter
import tensorflow as tf

#
loss = nn.CrossEntropyLoss(reduction='none')

input = torch.randn((20, 30, 30,2), requires_grad=True)
target = torch.empty((20,30,30), dtype=torch.long).random_(1)

logits= input.detach().numpy()
labels= target.detach().numpy()
input= input.permute(0,3,1,2)

output = loss(input, target)

def log_loss(logits,labels,name):
    SE_loss = tf.nn.sparse_softmax_cross_entropy_with_logits
    return tf.reduce_sum(SE_loss(logits=logits, labels=labels), [1, 2], name=name)
import sys
graph = tf.Graph()
with tf.Session(graph=graph) as sess:

    output_tf= log_loss(tf.convert_to_tensor(logits, dtype=tf.float32),tf.convert_to_tensor(labels, dtype=tf.int32), name="loss")
    # output_tf=tf.reduce_mean(output_tf)#uncomment this to get reduced answer for tensorflow
    print("*"*50)
    print("tensorflow: ",sess.run(tf.shape(output_tf)))
    print("Pytorch: ",output.shape)
    print("*"*50)#

Output of the code is given below which shows the dimensions or two cross entropy function outputs.

**************************************************
tensorflow:  [20]
Pytorch:  torch.Size([20, 30, 30])
**************************************************

If seems TF reduced the loss, so you could either remove reduction='none' or set it to your desired reduction type. :wink: