Dropout for long tensor?

1

Currently it prompts "fused_dropout" not implemented for 'Long', whereas it’s not a problem for Theano.

Is this feature we should wait for a while? or is it a bug?

2

Neither.
The problem is that modern dropout scales the outputs by 1/p. This would mean that you can only use 1/n, n integer . Also you don’t get autograd with long tensors. As such, the use-case seems too limited to include it in PyTorch given that torch.empty_like(a).bernoulli_(keep_prob) * a will give you an (unscaled) dropout for long with not much code.

Best regards

Thomas

3

There’s no much sense to do mandatory scaling in Dropout, this feature should be optional controlled by a input argument.

Anyway, thanks for replying.

4

I’m not so sure about this claim, as the expected values will be off, which will most likely result in bad validation and test performance as described in the Dropout paper.

1 Like