why does it drop sometimes more, sometimes less than 3 elements in a 2x5 matrix, when I have given probability 0.3
y = torch.ones(2, 5)
x = nn.Dropout(0.3)
x(y)
if got this output
tensor([[1.4286, 1.4286, 1.4286, 0.0000, 0.0000],
[1.4286, 1.4286, 1.4286, 1.4286, 1.4286]])
on later execution, this output
tensor([[0.0000, 0.0000, 1.4286, 1.4286, 0.0000],
[1.4286, 0.0000, 0.0000, 1.4286, 1.4286]])
Hi,
Each unit is dropped with a probability 0.3. In your case, you have 10 values. You may expect exactly 3 nodes to be dropped every time. It does not need to be exactly 3. Sometimes, it might be 2. Sometimes it can be 4 or even 5. I can one intuitive example.
Suppose you have an unbiased coin. P(H)=P(T)=0.5 (Here H-Head, T-Tail). If you toss the coin 10 times, on average you get 5 Heads and 5 Tails. But it can be 4 Heads and 6 Tails. The probability enforces that the expected number of heads in 10 tosses is 5.
Thanks
Regards
Pranavan
so what do I need to do to drop exactly 3 of 10 elements, i.e. I want to drop exactly 3 elements every time?
not more, not less.
In neural network, the randomness that we have is based on the probability. As far as I know, we do not have any modules which can enforce an exact behaviour. If you want the exact behaviour, you can select 3 indices at random and you can make it as zero. In that case, you have to write custom layers or modules in pytorch which extend the class nn.Module. Then you can plug it as a layer, in your code body.
Thanks