Thank you @cy-xu and @tom for your comments.
For now, I detached the result of the upsampling from the computation graph to exclude it from being used in the gradient computation. However, I added a loss function at that point in the network to provide some gradient for low layers. I want to keep the map intact and do an upscaling right away in one operation (see here other ways that I tried).
I known this is not even close to be considered as a solution, but I have to get rid of the non-determinism of the upsamling. My model relies heavily on the upsampled maps. Since they are non-deterministic, they make the final results totally unstable (useless!!!) within two runs where everything is deterministic except the upsampling function.
The same behavior is present in upscaling functions in other neural networks libraries (Chainer for instance).