Hi, all!
I try to use detach_() to truncate the gradient of a layer so that it does not propagate back. However, I find that the previous layer still have gradient update.
My goal is to make some of the convolution kernels in some layers not work, and not to participate in gradient back propagation of other layers. It sounds like dropout, but I don’t care if the parameter updates or not for these convolution kernels, because I always set them to 0 for a certain period of time.
I tried to truncate some kernels’ back propagation but still failed.And a similar question looks it is hard. If the gradient of the next layer(closer to output) doesn’t affect the gradient of the previous layer(closer to input), I will not consider this annoying problem.
Thanks in advance.
Hello, from here it seems that you are confused about intermediate value (non-leaf node) and weight of layers (leaf node), by .detach() the weight of layer would not have any effect on the gradiant flow back to the previous layers. If you want, you could .detach() on the intermediate value and the computation graph will break down.