Softmax Error Message Doesn't Make Sense

#1

recently, i’ve been seeing warnings saying that you need to add a ‘dim’ argument to Softmax as the implicit dimension selection is being deprecated.

I have a model that I found on github that uses a softmax layer (nn.LogSoftmax) in its forward function and an F.softmax() in its inference functions.

The data i’m feeding in has dimensions batch_size x output_classes. So, I replaced the call to F.softmax with F.softmax(___, dim=2)

This results in the very confusing and nonsensical error:

RuntimeError: Dimension out of range (expected to be in range of [-2, 1], but got 2)

I don’t … get why I would ever try to take a softmax over a negative dimension. Additionally, I don’t understand why this is stopping me from taking a softmax over the second dimension, when that is actually the dimension that I want to take the softmax over.

I tried swapping out the call to F.softmax with a call to self.softmax, which refers to the nn.LogSoftmax layer initialized in the model’s init() function with dim=2 as an argument, and got the same error, despite this softmax working during training.

I don’t understand what these errors mean, they seem totally illogical, and I don’t understand what the difference is between an nn.LogSoftmax and F.softmax in the first place.

#2

Could you print the shape of the tensor you are passing to F.softmax?
It seems dim2 is just missing. Are you adding the batch dimension to your data in the test case? This is often forgotten and yields these kind of errors.

You can call functions like softmax on “negative” dimensions to use reverse indices just like with python lists. -1 for example would be the last dimension.

The difference between nn.Softmax and F.softmax is not that big, as neither has any parameters stored. While nn.Softmax is an nn.Module and must therefore be initialized, F.softmax is just the functional API equivalent.

1 Like
#3

awesome!! you are definitely my hero today haha.

Input Dimension:

torch.Size([64, 5000])

Are dimensions then, by the same logic 0 indexed? Is the first dimension dimension 0?

1 Like
#4

yes. the dimensions are 0-indexed.