Does Dropout 'shortcut' if dropout is 0?

1

Two related questions:

  • if I create an nn.Dropout(0), does this get ‘shortcutted’ out, to become essentially an Identity function, by an if statement inside F.dropout?
  • if it doesnt, what is the performance impact (if any) if I dont add such if dropout > 0: guards myself?

(edit: since currently my code is filled with such guards :stuck_out_tongue: and I’m thinking of removing them)

3

(briefly thought I’d found the source, but that was for alpha_dropout, source for _functions.dropout.Dropout.apply seems elsewhere)

4

empirically, seems potentially shortcutted, eg run:

import torch
from torch import nn
import time


dropout = 0.01
layers = []
for i in range(10000):
    layers.append(nn.Dropout(dropout))

x = torch.rand(10000, requires_grad=True)
for l in layers:
    x = l(x)

back_start = time.time()
x.sum().backward()
print('back time', time.time() - back_start)

result for dropout=0:

back time 0.02752685546875

for dropout=0.01:

back time 0.11877131462097168

Additional evidence:

  • putting inplace=True fails if dropout is not 0
  • but works ok with dropout = 0
  • in addition backwards time is unchanged whether inplace is True or not, for dropout 0
1 Like
5

I think this is the implementation of the nn Module for Dropout. And so yes this check is already done.

5 Likes
6

(note: oh, the source code is in api folder. Interesting :slight_smile: I keep ending up like at https://github.com/pytorch/pytorch/tree/5d474e1812b2343b7dc1c9561f5f154334ccae38/torch/csrc/nn , and then I’ve no idea where to go next :slight_smile: )