**-- how to escape if synchronization. it is a huge problem because the evaluation of a simple if ONLY takes .33 sec while the entire forward in large network takes .0001s. here is a way that leads to wrong results because of synch gpu-to-cpu with non-blocking=True issue !!! WARNING: NEVER use gpu-to-cpu transfer with non-blocking=True. it is NOT safe. see the previous post. **
hi,
it seems that for a cuda tensors a and b, even with 1 element each, the if control will create a synchronization point such as in this example:
if a == b:
return 0
it is not the condition a == b that creates the synchronization, it is the if. if seems to trigger a transfer of the result of the comparison to cpu in a blocking way making it similar to call torch.cuda.synchronize() before if.
we will see here 2 examples: when it is if that evaluates the expression, and second example where we prepare the boolean value ourselves.
the example below is taken from my ‘real’ code. creating a snippet example wont show this issue.
i tried a solution using non_blocking=True but it GIVES THE WRONG RESULTS. this happens because the non-blcoking=True transfer. tensors are created in cpu filled with 0 but the transfer didnt finished yet while the cpu has already evaluated the condition. the doc does not seem to mention this possibility nor the transfer from gpu-2-cpu: Returns a Tensor with the specified device and (optional) dtype. If dtype is None it is inferred to be self.dtype. When non_blocking, tries to convert asynchronously with respect to the host if possible, e.g., converting a CPU Tensor with pinned memory to a CUDA Tensor.
this issue of transfering data from gpu to cpu will happen implicitly, but in safe mode == non-blocking=False, when you run a python operation , i.e.cpu, over cuda tensors, such as control statement, print, casting (int, …), calling cuda_tensor.item(), …
1. ‘if’ evaluates the condition:
example:
x = x.detach() # no gradient. x device: cuda.
tx = time.perf_counter()
min_x = x.min()
max_x = x.max()
print('time min max {}'.format(time.perf_counter() - tx))
tx = time.perf_counter()
if min_x == max_x:
return min_x
print('simple test Z {}'.format(time.perf_counter() - tx))
Output:
time min max 0.0002352111041545868
simple test Z 0.3392159678041935
2. we evaluate the condition and provide it to ‘if’:
in this way, we avoid the synchronization by performing the transfer from gpu to cpu with non_blocking=True. the doc says, that when this variable is true, the transfer is done asynchronously if possible.
the worst scenario is that you will endup doing a synch if resulting condition tensor is not ready.
but the issue, tensors on cpu are more likely to be wrong… because the copy is not finished yet while the cpu has already performed the evaluation…
example:
tx = time.perf_counter()
min_x = x.min()
max_x = x.max()
print('time min max {}'.format(time.perf_counter() - tx))
tx = time.perf_counter()
z = ((min_x - max_x) == 0).to(torch.device("cpu"), non_blocking=True)
print('compute Z {}'.format(time.perf_counter() - tx))
if z:
return min_x
print('simple test Z {}'.format(time.perf_counter() - tx))
Output:
time min max 0.00020601600408554077
compute Z 0.0007294714450836182
simple test Z 1.7356127500534058e-05
we get back to square one if we set non_blocking to false:
tx = time.perf_counter()
min_x = x.min()
max_x = x.max()
print('time min max {}'.format(time.perf_counter() - tx))
if ((min_x - max_x) == 0).to(torch.device("cpu"), non_blocking=False):
return min_x
print('simple test Z {}'.format(time.perf_counter() - tx))
Output:
otsu time min max 0.00021892786026000977
otsu simple test Z 0.3317955397069454
3. you wont see this behavior in this snippet:
import time
import torch
if __name__ == '__main__':
seed = 0
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
a = torch.rand(200, 200).to(device)
min_ = a.min()
max_ = a.max()
t = time.perf_counter()
if min_ == max_:
pass
print('time {}'.format(time.perf_counter() - t))
thanks