Cuda Compute Capability Issue on AWS EC2 p2.xlarge: GAN Dissection

  1. Trying to run GAN dissection from here: https://github.com/CSAILVision/gandissect.
  2. I followed the steps as mentioned, on an AWS Deep Learning AMI v19 that comes pre-configured with Anaconda and CUDA 9.0.
  3. The script on the github essentially does this:
    a. Creates a new Conda environment
    b. Installs a bunch of dependencies, including cudatoolkit, nccl and pytorch 0.4.1 (with GPU support)
    c. Installs their ‘netdissect’ package
    I followed their commands as listed on github, viz.:
script/setup_env.sh      # Create a conda environment with dependencies
script/download_data.sh  # Download support data and demo GANs
source activate netd     # Enter the conda environment
pip install -v -e .      # Link the local netdissect package into the env

Followed by:

python -m netdissect \
   --gan \
   --model "netdissect.proggan.from_pth_file('models/karras/livingroom_lsun.pth')" \
   --outdir "dissect/livingroom" \
   --layer layer1 layer4 layer7 \
   --size 1000

On a p2.xlarge instance (with Nvidia Tesla K80, compute capability 3.7) - the dissection FAILS as:

"cudaCheckError() failed : no kernel image is available for execution on the device"

However, on a p3.2xlarge instance type (with Nvidia Tesla V100, compute capability 7.0), it SUCCEEDS:

(netd) ubuntu@ip-172-31-44-139:~/gandissect$ python -m netdissect    --gan    --model "netdissect.proggan.from_pth_file('models/karras/livingroom_lsun.pth')"    --outdir "dissect/livingroom"    --layer layer1 layer4 layer7    --size 1000
Quantiles: 100%|###########################################################################################################################################################################################################################################################################################################################################| 10/10 [00:07<00:00,  1.30it/s]
label fracs: 100%|#####################################################################################################################################################################################################################################################################################################################################| 1000/1000 [01:58<00:00,  8.46it/s]
CQ layers:  67%|#############################################################################################################################################################################################################################3                                                                                                              | 2/3 [28:23<14:11, 851.90s/it]
Condquant:   5%|#################7                                                                                                                                                                                                                                                                                                                       | 54/1000 [00:34<10:02,  1.57it/s]
Condquant:   7%|######################7

I mention the Cuda capabilities because I’m unsure about the capability support included with PyTorch 0.4.1. The issue seems to be related to this.

Does anyone have an idea about what the issue could be?

Is there a way to find out what Cuda Compute Capabilities does a PyTorch installation support?

TIA.