I ask because I’ve heard you need pinned target memory for asynchronous copies and that for some reason pytorch might not be allocating pinned target tensors for GPU to CPU transfers with non_blocking=True? Yet I find it counter intuitive that gpu_tensor.to(‘cpu’, non_blocking=True) must always block… What is the truth?
Yes, as described in this section of the “A guide on good usage of non_blocking and pin_memory() in PyTorch” tutorial.