Softmax 2d doesn't make any sense, what am I missing?

1

Hi, I’m trying to use softamx2d and I can’t see what I’m doing wrong.
I will show my problem using something that will be easier to understand.
I have this 2d matrix of values and I want to make her to a probabilities matrix:

image
image1096×892 155 KB

so I’m using this code:

    self.softmax=nn.Softmax2d()

    result = self.softmax(result)

But I’m getting this result, all 0…, take a look:

image
image1131×878 56.3 KB

I can’t understand why? my “sanity-check” is that the sum of element of the entire matrix suppose to sum to 1 (I tried to make the precision bigger, no luck).
I know I’m missing something, can’t understand what.

Please help, Thanks!

2

Could you post the shape of your matrix?
nn.Softmax2d should apply the softmax using the channel dimension.

You could try the following code:

x = torch.randn(1, 1, 16, 16)
y = nn.Softmax(2)(x.view(1, 1, -1)).view_as(x)
1 Like
3

I will try this code.
My input shape is :(1,12,16,16), I want each “channel” to get softmax alone.

Edit: tried => y = nn.Softmax(2)(x.view(1, 1, -1)).view_as(x)
Got the same result :confused:

4

In that case, your code should be fine:

x = torch.randn(1, 12, 16, 16)
y = nn.Softmax2d()(x)
print(y.sum(1))

The sum over all channels will be 1 for each pixel position.

5

But I don’t want across channels, because they are not really channels, I want for each 2d 16x16 to have a softmax alone.means that each 16x16 will sum to 1.

6

Ah sorry, I misunderstood your use case.
My first code should work then:

y = nn.Softmax(2)(x.view(*x.size()[:2], -1)).view_as(x)
print(y[0, 1].sum())

Are you sure it’s not working?

2 Likes
7

I think it worked,but isn’t y[0,1] is the first two dimensions of y which are (1,12), and not (16,16) ,which those the ones I would like.
another weird thing is that what I got :

image
image1188×880 47.7 KB

In the first image in the first messege you can see the values before the softmax , isn’t that weird that there is only 1 value bigger then 0? maybe this is the biggest value, but there were some big values like:109/104/101 etc…Isn’t that weird!?

8

No, y[0, 1].shape will return a 16x16 tensor, so this should be fine.
Also, that’s not really weird but expected, as nn.Softmax()(torch.tensor([130., 109., 104.])) will give you a almost a 1 for the logit of 130. The difference between the logits is just large.
Have a look at the manual implementation:

torch.exp(x - x.max()) / torch.exp(x - x.max()).sum()
1 Like
9

Ok, got it , thanks a lot!

10

hey , i would like to apply softmax2d on different way
i have a tensor image shape of ‘[ batch_size , channel , width , height ]’ i transformed using FFT into a tensor of shape (batch_size, num_channels, num_freq, num_time)

‘Signal_filter.shape torch.Size([1, 1, 129, 256])’

softmax = nn.Softmax2d()
x = softmax(x.view(batch_size*num_channels, 1, num_freq, num_time)).view(batch_size, num_channels, num_freq, num_time)

i got this error :
"softmax_kernel_impl" not implemented for 'ComplexFloat'
11

You might need to apply the softmax operation on separate parts of the tensor or calculate e.g. the magnitude of the tensor. Alternatively, you could also check if a manual softmax implementation would work.

12

you were right thank you here the update code i did

mport torch
from torch.nn import functional as F


def complex_softmax(input, dim=None):
    """Applies the complex softmax function to an input tensor.
    """
    B , C , Freq,time_d = input.size()

    real = F.softmax(input.real, dim=dim)
    imaginary = F.softmax(input.imag, dim=dim)
    magnitudes = torch.sqrt((real ** 2 + imaginary ** 2))
    return (real * magnitudes).view(B*C ,Freq,time_d) * (
        imaginary * magnitudes
    ).view(B,C ,Freq,time_d)