I am using pytorch 1.0.0 and 1.0.1.post2 on ubuntu 16.04
I use 5d tensor for grid_sample, my code is like below
for data in dataloader: heatmap = model(data) grid_flow = f(data) new_heatmap = F.grid_sample(heatmap, grid_flow)
this code run smoothly on most of the data, but at certain data(3 out of about 1500), it will cause:
Traceback (most recent call last): File "..........py", line 401, in <module> cam_R = meta['camera_R'].to(dev) RuntimeError: CUDA error: an illegal memory access was encountered
I have already set
CUDA_LAUNCH_BLOCKING to ‘1’, but it just cannot catch the right place where the error happens.
And I also tried comment
new_heatmap = F.grid_sample(heatmap, grid_flow), there will be no error.
So, I am sure that the error occurred in F.grid_sample.
I change grid_sample mode to nearest,
new_heatmap = F.grid_sample(heatmap, grid_flow, mode='nearest') there is no runtime error.
I just cannot determine what may cause this runtime error,
Any idea or help will be appreciated!