THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1532576128691/work/aten/src/THC/THCCachingAllocator.cpp line=507 error=3 : initialization error

Traceback (most recent call last):
File “GeResult_ensemble.py”, line 70, in
GeResult()
File “GeResult_ensemble.py”, line 56, in GeResult
for (Input_O, Input_H, visit_tensor, anos) in Dataloader:
File “/home/omnisky/anaconda3/lib/python3.5/site-packages/torch/utils/data/dataloader.py”, line 336, in next
return self._process_next_batch(batch)
File “/home/omnisky/anaconda3/lib/python3.5/site-packages/torch/utils/data/dataloader.py”, line 357, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
RuntimeError: Traceback (most recent call last):
File “/home/omnisky/anaconda3/lib/python3.5/site-packages/torch/utils/data/dataloader.py”, line 106, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File “/home/omnisky/anaconda3/lib/python3.5/site-packages/torch/utils/data/dataloader.py”, line 106, in
samples = collate_fn([dataset[i] for i in batch_indices])
File “/media/RT_4TB/clj/2019Baidu-XJTU_URFC/2019BaiduXJTU/Dataloader/MultiModal_BDXJTU2019.py”, line 186, in getitem
Img_O = self.normal(Img)
File “/home/omnisky/anaconda3/lib/python3.5/site-packages/torchvision/transforms/transforms.py”, line 60, in call
img = t(img)
File “/home/omnisky/anaconda3/lib/python3.5/site-packages/torchvision/transforms/transforms.py”, line 163, in call
return F.normalize(tensor, self.mean, self.std, self.inplace)
File “/home/omnisky/anaconda3/lib/python3.5/site-packages/torchvision/transforms/functional.py”, line 206, in normalize
mean = torch.tensor(mean, dtype=torch.float32)
RuntimeError: cuda runtime error (3) : initialization error at /opt/conda/conda-bld/pytorch_1532576128691/work/aten/src/THC/THCCachingAllocator.cpp:507

Are you using multiprocessing (multiple workers in your DataLoader) and CUDA tensors?
If so, this post might be the answer.