envs:
NVIDIA GeForce RTX 4090
torch==1.10.0 torchvision==0.11.0
cuda 11.3
(kornia==0.7.1)
error:
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [32,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [33,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [34,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [35,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [36,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [37,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [38,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [39,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [40,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [41,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [42,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [43,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [44,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [45,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [46,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [47,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [48,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [49,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [50,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [51,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [52,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [53,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [54,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [55,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [56,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [57,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [58,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [59,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [60,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [61,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [62,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [63,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
Traceback (most recent call last):
File "train_cifar10.py", line 401, in <module>
main()
File "train_cifar10.py", line 125, in main
main_worker(args.gpu, ngpus_per_node, args)
File "train_cifar10.py", line 212, in main_worker
train(model, optimizer, criterion, train_loader, train_sampler, test_loader, args, ngpus_per_node)
File "train_cifar10.py", line 310, in train
train_ep(args, model, train_loader, composite_attack, optimizer, criterion)
File "train_cifar10.py", line 243, in train_ep
data_adv = composite_attack(data_adv, target)
File "/home/wtd/miniconda3/envs/py37-113-t100/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/wtd/composite-adv-main/composite_adv/composite_attack.py", line 132, in forward
return self.caa_attack(inputs, labels)
File "/home/wtd/composite-adv-main/composite_adv/composite_attack.py", line 323, in caa_attack
self.update_attack_order(images, labels, adv_val)
File "/home/wtd/composite-adv-main/composite_adv/composite_attack.py", line 281, in update_attack_order
_adv_img, _ = self.attack_dict[idx](prev_img, adv_val[idx], labels)
File "/home/wtd/composite-adv-main/composite_adv/composite_attack.py", line 181, in caa_saturation
ori_is_attacked=self.is_attacked.clone())
File "/home/wtd/composite-adv-main/composite_adv/composite_attack.py", line 139, in _comp_pgd
return self.attack_pool_base[attack_idx](data, attack_parameter), attack_parameter
File "/home/wtd/miniconda3/envs/py37-113-t100/lib/python3.7/site-packages/kornia/enhance/adjust.py", line 137, in adjust_saturation
out: Tensor = hsv_to_rgb(x_adjusted)
File "/home/wtd/miniconda3/envs/py37-113-t100/lib/python3.7/site-packages/kornia/color/hsv.py", line 85, in hsv_to_rgb
one: torch.Tensor = torch.tensor(1.0, device=image.device, dtype=image.dtype)
RuntimeError: CUDA error: device-side assert triggered
other information:
This crash appears in the following short loop,but I believe there is no problem with this loop.Furthermore,in this loop, when outputting a variable with its address and content unchanged, the first few attempts can proceed normally, but a sudden crash may occur in the middle, resulting in the aforementioned error.Here is the detailed loop:
for tdx in range(self.seq_num):
prev_img = adv_img.clone()
adv_img = torch.zeros_like(adv_img)
for idx in range(self.seq_num):
print("idx:",idx)
print("id:",id(adv_val))
print("adv_val[0]:",adv_val[0])
# In the next line, adv_val.detach().clone() is done immediately after the adv_val
# is passed into the attack_dict[0] function
_adv_img, _ = self.attack_dict[0](prev_img, adv_val[0], labels)
adv_img = adv_img + self.curr_dsm[tdx][idx] * _adv_img
the final output before the error:
idx: 0
id: 139993589367984
adv_val[0]: tensor([ 1.4882, 2.8088, -1.2813, -2.2428, -1.9623, 0.0631, -1.8943, 2.0455],
device='cuda:0')
idx: 1
id: 139993589367984
adv_val[0]: tensor([ 1.4882, 2.8088, -1.2813, -2.2428, -1.9623, 0.0631, -1.8943, 2.0455],
device='cuda:0')
idx: 0
id: 139993589367984
adv_val[0]: tensor([ 1.4882, 2.8088, -1.2813, -2.2428, -1.9623, 0.0631, -1.8943, 2.0455],
device='cuda:0')
idx: 1
id: 139993589367984
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [84,0,0], thread: [32,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
Contrary to expectations, crash suddenly occurred during the printing process of adv_val[0](don’t print it at last).I think it’s very strange to crash when printing this variable with both the address and content unchanged.
I would greatly appreciate it if you could provide suggestions to solve this problem. If there are any descriptions I don’t describe clearly, please ask me.