I get this error - Please let me know if you may have any suggestions?
(gdrnpp) mona@ada:~/gdrnpp_bop2022$ ./det/yolox/tools/test_yolox.sh ./configs/yolox/bop_pbr/yolox_x_640_augCozyAAEhsv_ranger_30_epochs_mona_bop_test.py 0 ./output/yolox/bop_pbr/yolox_x_640_augCozyAAEhsv_ranger_30_epochs_mona_bop_test/model_final.pth
_module.pnp_net.features.0.weight
_module.pnp_net.features.1.{bias, weight}
_module.pnp_net.features.3.weight
_module.pnp_net.features.4.{bias, weight}
_module.pnp_net.features.6.weight
_module.pnp_net.features.7.{bias, weight}
_module.pnp_net.fc1.{bias, weight}
_module.pnp_net.fc2.{bias, weight}
_module.pnp_net.fc_r.{bias, weight}
_module.pnp_net.fc_t.{bias, weight}
[0207_134552 detectron2@57]: Fusing conv bn...
ERROR [0207_134553 d2.engine.launch@82]: An error has been caught in function 'launch', process 'MainProcess' (839409), thread 'MainThread' (140334550648640):
Traceback (most recent call last):
File "/home/mona/gdrnpp_bop2022/./det/yolox/tools/main_yolox.py", line 70, in <module>
launch(
-> <function launch at 0x7fa2132cac10>
> File "/home/mona/anaconda3/envs/gdrnpp/lib/python3.9/site-packages/detectron2/engine/launch.py", line 82, in launch
main_func(*args)
| -> (Namespace(config_file='./configs/yolox/bop_pbr/yolox_x_640_augCozyAAEhsv_ranger_30_epochs_mona_bop_test.py', resume=False,...
-> <function main at 0x7fa0a6bc98b0>
File "/home/mona/gdrnpp_bop2022/./det/yolox/tools/main_yolox.py", line 58, in main
model = fuse_model(model)
| -> YOLOX(
| (backbone): YOLOPAFPN(
| (backbone): CSPDarknet(
| (stem): Focus(
| (conv): BaseConv(
| (conv): ...
-> <function fuse_model at 0x7fa113d48310>
File "/home/mona/gdrnpp_bop2022/det/yolox/tools/../../../det/yolox/utils/model_utils.py", line 67, in fuse_model
m.conv = fuse_conv_and_bn(m.conv, m.bn) # update conv
| | | -> BaseConv(
| | | (conv): Conv2d(12, 80, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
| | | (bn): BatchNorm2d(80, eps...
| | -> BaseConv(
| | (conv): Conv2d(12, 80, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
| | (bn): BatchNorm2d(80, eps...
| -> <function fuse_conv_and_bn at 0x7fa113d48280>
-> BaseConv(
(conv): Conv2d(12, 80, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(80, eps...
File "/home/mona/gdrnpp_bop2022/det/yolox/tools/../../../det/yolox/utils/model_utils.py", line 57, in fuse_conv_and_bn
fusedconv.bias.copy_(torch.mm(w_bn, b_conv.reshape(-1, 1)).reshape(-1) + b_bn)
| | | | | | -> tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
| | | | | | 0., 0., 0., 0...
| | | | | -> <method 'reshape' of 'torch._C._TensorBase' objects>
| | | | -> tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
| | | | 0., 0., 0., 0...
| | | -> tensor([[0.9995, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000],
| | | [0.0000, 0.9995, 0.0000, ..., 0.0000, 0.0000, 0.0000...
| | -> <built-in method mm of type object at 0x7fa1e1695ee0>
| -> <module 'torch' from '/home/mona/anaconda3/envs/gdrnpp/lib/python3.9/site-packages/torch/__init__.py'>
-> Conv2d(12, 80, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`
and I have:
(gdrnpp) mona@ada:~/gdrnpp_bop2022$ python -m torch.utils.collect_env
Collecting environment information...
PyTorch version: 1.10.1
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A
OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: 14.0.0-1ubuntu1.1
CMake version: version 3.22.1
Libc version: glibc-2.35
Python version: 3.9.18 (main, Sep 11 2023, 13:41:44) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-6.2.0-39-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.8.89
GPU models and configuration: GPU 0: NVIDIA RTX 6000 Ada Generation
Nvidia driver version: 535.104.12
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.7
HIP runtime version: N/A
MIOpen runtime version: N/A
Versions of relevant libraries:
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.3
[pip3] pytorch-lightning==1.6.0
[pip3] torch==1.10.1
[pip3] torchaudio==0.10.1
[pip3] torchmetrics==1.3.0.post0
[pip3] torchvision==0.11.2
[conda] blas 1.0 mkl conda-forge
[conda] cudatoolkit 11.3.1 hb98b00a_12 conda-forge
[conda] ffmpeg 4.3 hf484d3e_0 pytorch
[conda] mkl 2023.1.0 h213fc3f_46344
[conda] mypy-extensions 1.0.0 pypi_0 pypi
[conda] numpy 1.26.3 py39h474f0d3_0 conda-forge
[conda] pytorch 1.10.1 py3.9_cuda11.3_cudnn8.2.0_0 pytorch
[conda] pytorch-lightning 1.6.0 pypi_0 pypi
[conda] pytorch-mutex 1.0 cuda pytorch
[conda] torch 2.2.0 pypi_0 pypi
[conda] torchaudio 0.10.1 py39_cu113 pytorch
[conda] torchmetrics 1.3.0.post0 pypi_0 pypi
[conda] torchvision 0.11.2 py39_cu113 pytorch
I am using this repo