CUDA, cuDNN, and PyTorch Compatibility

1

Hello,

I’m trying to set up a specific environment on my university’s HPC, which restricts sudo access. The HPC has Python >=3.9 and CUDA >=11.7. For my project, I need Python 3.6 and PyTorch 0.4.1, compatible with CUDA 9.2 and cuDNN 7.2.1.

What I’ve done:

  1. Created a conda environment with Python 3.6.
  2. Installed cudatoolkit=9.2 and cudnn=7.2.1.
  3. Installed PyTorch 0.4.1 using conda install pytorch=0.4.1 cuda92 -c pytorch.

Issues:

  • When installing pytorch 0.4.1 in this env i got env conflicts, so i created a python venv inside the conda env and installed 0.4.1 using pip.
  • When running nvcc --version, it shows CUDA 9.2. Also torch.cuda.version returns 9.2 which is good.
  • torch.version.cuda shows 9.2, but torch.backends.cudnn.version() returns 7.1 instead of 7.2.1.
  • During training, I encounter the error: RuntimeError: CuDNN error: CUDNN_STATUS_EXECUTION_FAILED.

Questions:

  1. Why is cuDNN version 7.1 instead of 7.2.1?
  2. How can I correctly set up the environment to avoid conflicts?
  3. How to solve CUDNN error?

Thank you for your help!

2

What’s the reason for using such an old PyTorch release, which is approx. 6 years old?

3

I was trying to use a framework called MedicalDetectionToolkit. GitHub - MIC-DKFZ/medicaldetectiontoolkit: The Medical Detection Toolkit contains 2D + 3D implementations of prevalent object detectors such as Mask R-CNN, Retina Net, Retina U-Net, as well as a training and inference framework focused on dealing with medical images.
It said that it needs python3.6.
Also the cuda functions there are implementeted for python3.6

4

I would use this instead. Doesn’t support RCNN at the moment i believe. Same group.

5

Yes this is for 3d data only. I am working on 2d data @Soumya_Kundu

6

I think you will find it much easier implementing the 2d versions yourself from scratch. There are also great repos for all them as well. Especially Mask-RCNN.

7

Hi @Soumya_Kundu , is there any repo you know that is working on this from scratch. What i wanted was a algorithm that specially targets on medical images. I have tried RetinaNet and Yolo. The result from RetinaNet was good but, i wanted to improve the result by using the model specific to the domain. So i tried MDT using retinaUnet.

8

What exactly are you searching for? 2D Detection networks for medical images?

9

yes,
i am working at LVO detection and used various networks. They work good but i want to improve the result. My thoughts are that if the medical images based network will work better in this case.
MDT RetinaUnet and nnDetection were the option but MDT is too old and nnDetection implementation is for 3D only. I was just searching if there is some implementation for 2D.

10

RetinaUNet is for segmentation right?

There is no “medical detection” framework that I know of unfortunately. I think you should take your best performing model and try improving that further by playing around with it a bit more.

11

Yes, but MedicalDetectionToolkit has used to for detection purpose by taking advantage of Segmentation Supervision of UNET.
Thank you @Soumya_Kundu for your time

12

I think that statement is a bit flawed. Most literature points to generic semantic segmentation models not being able to detect well.