Sampled softmax loss

beegii · February 2, 2017, 7:47am

Hi,

Does sampled softmax loss exist in pytorch? I cound not find it.

Thanks

smth · February 2, 2017, 8:46am

can you elaborate on what this is? with a link to an implementation elsewhere?
afaik this is not in pytorch core.

beegii · February 2, 2017, 8:53am

Thanks for your reply.
Tensorflow has this: https://www.tensorflow.org/api_docs/python/nn/candidate_sampling

apaszke · February 2, 2017, 9:11pm

We don’t have any such thing in the core. We’ll need to add them. Thanks for the pointer!

ngimel · February 3, 2017, 9:41pm

There’s a lua torch implementation

github.com

Element-Research/dpnn/blob/master/NCECriterion.lua

------------------------------------------------------------------------
--[[ Noise Contrast Estimation Criterion ]]--
-- Ref.: A. http://mi.eng.cam.ac.uk/~xc257/papers/ICASSP2015-rnnlm-nce.pdf
--       B. https://www.cs.toronto.edu/~amnih/papers/ncelm.pdf
------------------------------------------------------------------------
local NCECriterion, parent = torch.class("nn.NCECriterion", "nn.Criterion")
local eps = 0.0000001

function NCECriterion:__init()
   parent.__init(self)  
   self.sizeAverage = true
   
   self.gradInput = {torch.Tensor(), torch.Tensor(), torch.Tensor(), torch.Tensor()}   
end

function NCECriterion:updateOutput(inputTable, target)
   -- P_model(target), P_model(sample), P_noise(target), P_noise(sample)
   local Pmt, Pms, Pnt, Pns = unpack(inputTable)
   local k = Pms:size(2)

This file has been truncated. show original

beegii · February 3, 2017, 9:47pm

@ngimel: Thanks for the link. I don’t know Lua and will have a look at the code.

lopuhin · February 25, 2017, 1:35pm

As far as I know, NCE (Noise Contrast Estimation) is different from sampled softmax from tensorflow, see Jozefowicz et al. (2016) or here for a comparison.

EDIT: sorry, I see that original link is to a page with a number of different softmax approximations, and NCE is one of them. I personally would be more interested in sampled softmax, as it tends to work better for me.

EDIT2: here is a TF implementation of sampled softmax and NCE, hopefully they can be implemented using existing pytorch functions.

vince62s · February 27, 2017, 5:42pm

You may also be interested in this implementation:

Giving very good results for the LM task.

beegii · March 2, 2017, 7:04am

@lopuhin and @vince62s: thanks for your suggestions.

windweller · September 11, 2018, 3:54pm

Any updates on this?? It probably isn’t a priority…but I secretly wish PyTorch has a big development team like TensorFlow and can add these functionalities easily!

smth · September 11, 2018, 4:44pm

@windweller Adaptive Softmax was part of PyTorch 0.4.1, see: https://pytorch.org/docs/stable/nn.html#torch.nn.AdaptiveLogSoftmaxWithLoss

Sampled Softmax is implemented in this repo: https://github.com/rdspring1/PyTorch_GBW_LM

windweller · September 11, 2018, 5:50pm

Nice and incredible!! Will start using it now