Non-blocking device to host transfer

[GuillaumeTong]

Here is a related discussion FYR:

How to maximize CPU <==> GPU memory transfer speeds?

@ptrblck If I “strace -f” my GPU app I find not a single mlockxxx call nor a mmap with MAP_LOCKED call. I presume there isn’t a major bug in pytorch where they are missing doing the memory pin such that crashes might occur under stress if the system starts swapping pages. I only know of the two ways mentioned to pin memory pages.

Here is the TLDR:

1. gpu_tensor.to(device="cpu", non_blocking=True) will asynchronously copy a tensor from GPU to CPU pinned memory
2. As you point out, pinned_cpu_tensor.copy_(gpu_tensor, non_blocking=True) will asynchronously copy a tensor from GPU to an existing CPU pinned memory tensor
3. Once the asynchronous copy has been launched, you need to use torch.cuda.Stream.synchronize() or torch.cuda.synchronize() to ensure the copy has finished before using its content