# Sampled softmax loss **URL:** \ **Category:** Uncategorized\ **Created:** [February 2, 2017, 7:47am UTC](https://discuss.pytorch.org/t/sampled-softmax-loss/305 "2017-02-02T07:47:02Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![beegii](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/f14d63/32.png) [@beegii](https://discuss.pytorch.org/u/beegii)\ **Post date:** [February 2, 2017, 7:47am UTC](https://discuss.pytorch.org/t/sampled-softmax-loss/305/1 "2017-02-02T07:47:02Z")
Hi, Does sampled softmax loss exist in pytorch? I cound not find it. Thanks ---
**Author:** ![smth](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/32/13_2.png) [@smth](https://discuss.pytorch.org/u/smth)\ **Post date:** [February 2, 2017, 8:46am UTC](https://discuss.pytorch.org/t/sampled-softmax-loss/305/2 "2017-02-02T08:46:14Z")
can you elaborate on what this is? with a link to an implementation elsewhere? afaik this is not in pytorch core. ---
**Author:** ![beegii](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/f14d63/32.png) [@beegii](https://discuss.pytorch.org/u/beegii)\ **Post date:** [February 2, 2017, 8:53am UTC](https://discuss.pytorch.org/t/sampled-softmax-loss/305/3 "2017-02-02T08:53:20Z")
Thanks for your reply. Tensorflow has this: [https://www.tensorflow.org/api\_docs/python/nn/candidate\_sampling](https://www.tensorflow.org/api_docs/python/nn/candidate_sampling) ---
**Author:** ![apaszke](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apaszke/32/21_2.png) [@apaszke](https://discuss.pytorch.org/u/apaszke)\ **Post date:** [February 2, 2017, 9:11pm UTC](https://discuss.pytorch.org/t/sampled-softmax-loss/305/4 "2017-02-02T21:11:53Z")
We don’t have any such thing in the core. We’ll need to add them. Thanks for the pointer! ---
**Author:** ![ngimel](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ngimel/32/14766_2.png) [@ngimel](https://discuss.pytorch.org/u/ngimel)\ **Post date:** [February 3, 2017, 9:41pm UTC](https://discuss.pytorch.org/t/sampled-softmax-loss/305/5 "2017-02-03T21:41:50Z")
There’s a lua torch implementation > ---
**Author:** ![beegii](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/f14d63/32.png) [@beegii](https://discuss.pytorch.org/u/beegii)\ **Post date:** [February 3, 2017, 9:47pm UTC](https://discuss.pytorch.org/t/sampled-softmax-loss/305/6 "2017-02-03T21:47:17Z")
@ngimel: Thanks for the link. I don’t know Lua and will have a look at the code. ---
**Author:** ![lopuhin](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lopuhin/32/182_2.png) [@lopuhin](https://discuss.pytorch.org/u/lopuhin)\ **Post date:** [February 25, 2017, 1:35pm UTC](https://discuss.pytorch.org/t/sampled-softmax-loss/305/7 "2017-02-25T13:35:25Z")
As far as I know, NCE (Noise Contrast Estimation) is different from sampled softmax from tensorflow, see Jozefowicz et al. (2016) or [here](http://sebastianruder.com/word-embeddings-softmax/index.html#noisecontrastiveestimation) for a comparison. EDIT: sorry, I see that original link is to a page with a number of different softmax approximations, and NCE is one of them. I personally would be more interested in sampled softmax, as it tends to work better for me. EDIT2: [here](https://github.com/tensorflow/tensorflow/blob/131c3a67a7b8d27fd918e0bc5bddb3cb086de57e/tensorflow/python/ops/nn_impl.py) is a TF implementation of sampled softmax and NCE, hopefully they can be implemented using existing pytorch functions. ---
**Author:** ![vince62s](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vince62s/32/3661_2.png) [@vince62s](https://discuss.pytorch.org/u/vince62s)\ **Post date:** [February 27, 2017, 5:42pm UTC](https://discuss.pytorch.org/t/sampled-softmax-loss/305/8 "2017-02-27T17:42:34Z")
You may also be interested in this implementation: > **[facebookresearch/adaptive-softmax](https://github.com/facebookresearch/adaptive-softmax)** > > Implements an efficient softmax approximation as described in the paper "Efficient softmax approximation for GPUs" (http://arxiv.org/abs/1609.04309) - facebookresearch/adaptive-softmax Giving very good results for the LM task. ---
**Author:** ![beegii](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/f14d63/32.png) [@beegii](https://discuss.pytorch.org/u/beegii)\ **Post date:** [March 2, 2017, 7:04am UTC](https://discuss.pytorch.org/t/sampled-softmax-loss/305/9 "2017-03-02T07:04:01Z")
@lopuhin and @vince62s: thanks for your suggestions. ---
**Author:** ![windweller](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/windweller/32/1079_2.png) [@windweller](https://discuss.pytorch.org/u/windweller)\ **Post date:** [September 11, 2018, 3:54pm UTC](https://discuss.pytorch.org/t/sampled-softmax-loss/305/10 "2018-09-11T15:54:58Z")
Any updates on this?? It probably isn’t a priority…but I secretly wish PyTorch has a big development team like TensorFlow and can add these functionalities easily! ---
**Author:** ![smth](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/32/13_2.png) [@smth](https://discuss.pytorch.org/u/smth)\ **Post date:** [September 11, 2018, 4:44pm UTC](https://discuss.pytorch.org/t/sampled-softmax-loss/305/11 "2018-09-11T16:44:51Z")
@windweller Adaptive Softmax was part of PyTorch 0.4.1, see: [https://pytorch.org/docs/stable/nn.html#torch.nn.AdaptiveLogSoftmaxWithLoss](https://pytorch.org/docs/stable/nn.html#torch.nn.AdaptiveLogSoftmaxWithLoss) Sampled Softmax is implemented in this repo: [https://github.com/rdspring1/PyTorch\_GBW\_LM](https://github.com/rdspring1/PyTorch_GBW_LM) ---
**Author:** ![windweller](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/windweller/32/1079_2.png) [@windweller](https://discuss.pytorch.org/u/windweller)\ **Post date:** [September 11, 2018, 5:50pm UTC](https://discuss.pytorch.org/t/sampled-softmax-loss/305/12 "2018-09-11T17:50:45Z")
Nice and incredible!! Will start using it now 🙂