how to apply Softmax to sparse Softmax function.
Pytorch doesn’t have a sparse softmax function. However, you could work with the sparse matrix’s indices and values to do this. Let tensor be your sparse matrix. There are two main things you want to do:
- Get each row of the (sparse) tensor
- Apply softmax to each row
For 1), you can pick out the elements of tensor._indices() that have the same row by looking.
For 2), after you collect the relevant values of tensor._values() for each row, you can just call softmax on it.
1 Like
It dose fine works!
Thank you so much.
However, I know there are differences between value from dense applied to Softmax and value from sparse applied to Softmax, like below (image).
Then, tell me what you think about this.
Each of sum of row vector is 1.
So, which value is appropriate for representing information?
I think it is because when you transfer a matrix to dense, it will fill zero in other place rather than -inf.
So, zero will also get some probability when you do softmax.