Data-dependent custom dropout

1

I am new to ML and pytorch, so probably this is a dumb question.

I saw some posts to make a custom mask for dropout, but in my case, the mask depends on the input: drop out the data in a certain range, drop out max value, or drop out min value, etc. just to mention a few examples.

Since the mask-making process is not differentiable in general, I do not want the mask generation be a part of the backpropagation. So in forward pass, I do the mask generation based on the inputs and apply the mask. And in backprop, I only do the backprop for the mask apply without mask generation. Can I do this in pytorch? If so, how can I do this?

2

Do you mean something like:

    with torch.no_grad():
        mask = ... # mask generation code

You can then use the mask however you want. I’m actually curious if you need to explicitly use no_grad here if your mask generation code doesn’t rely on any weights…

3

Thanks! Will try this :slight_smile:

Regarding no_grad, IIUC it is required, otherwise the gradient from the output does not propagate to the input (but I am not sure though).