THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=405 error=11 : invalid argument
Traceback (most recent call last):
File “/home/rixi/lzx/WaveGlow/train.py”, line 188, in
train(num_gpus, args.rank, args.group_name, **train_config)
File “/home/rixi/lzx/WaveGlow/train.py”, line 125, in train
outputs = model((mel, audio))
File “/home/rixi/lzx/WaveGlow/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 489, in call
result = self.forward(*input, **kwargs)
File “/home/rixi/lzx/WaveGlow/glow.py”, line 233, in forward
audio, log_det_W = self.convinvk
File “/home/rixi/lzx/WaveGlow/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 489, in call
result = self.forward(*input, **kwargs)
File “/home/rixi/lzx/WaveGlow/glow.py”, line 100, in forward
log_det_W = batch_size * n_of_groups * torch.logdet(W)
RuntimeError: cublas runtime error : the GPU program failed to execute at /pytorch/aten/src/THC/THCBlas.cu:511
CUDA=10.0 TORCH=1.0