.cuda() freezes the python

Hi guys. I am a newbie in pytorch.

My Ubuntu 16.04.02 LTS got python2.7 and cuda 9.0 installed (torch.cuda.is_available() returns True). However, when I tried run the following, my python just got frozen and I can only type ctrl+z to force exit.

import torch
torch.randn(10).cuda()

Then freeze until ctrl+z.

Did I do something wrong? Anyone could help? Thanks a lot.

Hi, i see similar issue during use CUDA.

And I try to check GPU memory, it will goes up until ~229MB. Python then can work normally.

I am not sure why. If you find a way to fix it, please tell me. thanks

I’m facing the same issue.
I’m running on jupyter notebook, the first call to torch.randn(10).cuda() takes a long loooong time to complete (~7 min), but subsequent calls behave normally.

I’m posting a screenshoot of the stack when interrupting the kernel.

1 Like

What is the output of your nvidia-smi when this is happening?
Could you try building PyTorch from source and seeing if the long cuda() calls still occur?

This is the output of nvidia-smi (fastai/bin/python is the process related to the notebook):
image
I will try to build from source.

I couldn’t manage to build from source, first I was getting this error, I followed the workaround but then got an error regarding cuDNN version, my cuDNN installation is a mess because of tensorflow, so I didn’t wanted to mess with that afraid that it could mess my tensorflow installation. So I switched to docker, and the first run of torch.randn(10).cuda() is taking ~2 sec, which I think is expected.

@lgvaz you have a 1070 GPU, but you must’ve installed pytorch for CUDA 7.5. What it is doing is recompiling pytorch for your GPU (cuda8 supported GPU).
If you reinstall the CUDA8 version of pytorch, the freezing wont happen. Select the right option in our version selector on http://pytorch.org

2 Likes

Hi smth,

i also see the same issue even with CUDA8.

@Spencer_ML It’s possible that there’s a mismatch between the CUDA version your pytorch was compiled with, and the CUDA version you’re running your computer on. How did you install pytorch?

Thanks all for the discussion.
Have successfully tried cuda() but on the first run of .cuda(), I found that I have to wait for 5 to 10 minutes. Is it normal or should I do some config first?

Can you run the following and report the output?

@QuantScientist Sure. Here are my outputs:
##############################################
##########01 PyCUDA verify CUDA 8.0.ipynb########

1 device(s) found.
Device #0: GeForce GTX 1080 Ti
Compute Capability: 6.1
Total Memory: 11440512 KB
ASYNC_ENGINE_COUNT: 2
CAN_MAP_HOST_MEMORY: 1
CLOCK_RATE: 1582000
COMPUTE_CAPABILITY_MAJOR: 6
COMPUTE_CAPABILITY_MINOR: 1
COMPUTE_MODE: DEFAULT
CONCURRENT_KERNELS: 1
ECC_ENABLED: 0
GLOBAL_L1_CACHE_SUPPORTED: 1
GLOBAL_MEMORY_BUS_WIDTH: 352
GPU_OVERLAP: 1
INTEGRATED: 0
KERNEL_EXEC_TIMEOUT: 1
L2_CACHE_SIZE: 2883584
LOCAL_L1_CACHE_SUPPORTED: 1
MANAGED_MEMORY: 1
MAXIMUM_SURFACE1D_LAYERED_LAYERS: 2048
MAXIMUM_SURFACE1D_LAYERED_WIDTH: 32768
MAXIMUM_SURFACE1D_WIDTH: 32768
MAXIMUM_SURFACE2D_HEIGHT: 65536
MAXIMUM_SURFACE2D_LAYERED_HEIGHT: 32768
MAXIMUM_SURFACE2D_LAYERED_LAYERS: 2048
MAXIMUM_SURFACE2D_LAYERED_WIDTH: 32768
MAXIMUM_SURFACE2D_WIDTH: 131072
MAXIMUM_SURFACE3D_DEPTH: 16384
MAXIMUM_SURFACE3D_HEIGHT: 16384
MAXIMUM_SURFACE3D_WIDTH: 16384
MAXIMUM_SURFACECUBEMAP_LAYERED_LAYERS: 2046
MAXIMUM_SURFACECUBEMAP_LAYERED_WIDTH: 32768
MAXIMUM_SURFACECUBEMAP_WIDTH: 32768
MAXIMUM_TEXTURE1D_LAYERED_LAYERS: 2048
MAXIMUM_TEXTURE1D_LAYERED_WIDTH: 32768
MAXIMUM_TEXTURE1D_LINEAR_WIDTH: 134217728
MAXIMUM_TEXTURE1D_MIPMAPPED_WIDTH: 16384
MAXIMUM_TEXTURE1D_WIDTH: 131072
MAXIMUM_TEXTURE2D_ARRAY_HEIGHT: 32768
MAXIMUM_TEXTURE2D_ARRAY_NUMSLICES: 2048
MAXIMUM_TEXTURE2D_ARRAY_WIDTH: 32768
MAXIMUM_TEXTURE2D_GATHER_HEIGHT: 32768
MAXIMUM_TEXTURE2D_GATHER_WIDTH: 32768
MAXIMUM_TEXTURE2D_HEIGHT: 65536
MAXIMUM_TEXTURE2D_LINEAR_HEIGHT: 65000
MAXIMUM_TEXTURE2D_LINEAR_PITCH: 2097120
MAXIMUM_TEXTURE2D_LINEAR_WIDTH: 131072
MAXIMUM_TEXTURE2D_MIPMAPPED_HEIGHT: 32768
MAXIMUM_TEXTURE2D_MIPMAPPED_WIDTH: 32768
MAXIMUM_TEXTURE2D_WIDTH: 131072
MAXIMUM_TEXTURE3D_DEPTH: 16384
MAXIMUM_TEXTURE3D_DEPTH_ALTERNATE: 32768
MAXIMUM_TEXTURE3D_HEIGHT: 16384
MAXIMUM_TEXTURE3D_HEIGHT_ALTERNATE: 8192
MAXIMUM_TEXTURE3D_WIDTH: 16384
MAXIMUM_TEXTURE3D_WIDTH_ALTERNATE: 8192
MAXIMUM_TEXTURECUBEMAP_LAYERED_LAYERS: 2046
MAXIMUM_TEXTURECUBEMAP_LAYERED_WIDTH: 32768
MAXIMUM_TEXTURECUBEMAP_WIDTH: 32768
MAX_BLOCK_DIM_X: 1024
MAX_BLOCK_DIM_Y: 1024
MAX_BLOCK_DIM_Z: 64
MAX_GRID_DIM_X: 2147483647
MAX_GRID_DIM_Y: 65535
MAX_GRID_DIM_Z: 65535
MAX_PITCH: 2147483647
MAX_REGISTERS_PER_BLOCK: 65536
MAX_REGISTERS_PER_MULTIPROCESSOR: 65536
MAX_SHARED_MEMORY_PER_BLOCK: 49152
MAX_SHARED_MEMORY_PER_MULTIPROCESSOR: 98304
MAX_THREADS_PER_BLOCK: 1024
MAX_THREADS_PER_MULTIPROCESSOR: 2048
MEMORY_CLOCK_RATE: 5505000
MULTIPROCESSOR_COUNT: 28
MULTI_GPU_BOARD: 0
MULTI_GPU_BOARD_GROUP_ID: 0
PCI_BUS_ID: 1
PCI_DEVICE_ID: 0
PCI_DOMAIN_ID: 0
STREAM_PRIORITIES_SUPPORTED: 1
SURFACE_ALIGNMENT: 512
TCC_DRIVER: 0
TEXTURE_ALIGNMENT: 512
TEXTURE_PITCH_ALIGNMENT: 32
TOTAL_CONSTANT_MEMORY: 65536
UNIFIED_ADDRESSING: 1
WARP_SIZE: 32

##############################################

Global memory occupancy:98.000000% free

===Attributes for device 0
MAX_THREADS_PER_BLOCK:1024
MAX_BLOCK_DIM_X:1024
MAX_BLOCK_DIM_Y:1024
MAX_BLOCK_DIM_Z:64
MAX_GRID_DIM_X:2147483647
MAX_GRID_DIM_Y:65535
MAX_GRID_DIM_Z:65535
MAX_SHARED_MEMORY_PER_BLOCK:49152
TOTAL_CONSTANT_MEMORY:65536
WARP_SIZE:32
MAX_PITCH:2147483647
MAX_REGISTERS_PER_BLOCK:65536
CLOCK_RATE:1582000
TEXTURE_ALIGNMENT:512
GPU_OVERLAP:1
MULTIPROCESSOR_COUNT:28
KERNEL_EXEC_TIMEOUT:1
INTEGRATED:0
CAN_MAP_HOST_MEMORY:1
COMPUTE_MODE:DEFAULT
MAXIMUM_TEXTURE1D_WIDTH:131072
MAXIMUM_TEXTURE2D_WIDTH:131072
MAXIMUM_TEXTURE2D_HEIGHT:65536
MAXIMUM_TEXTURE3D_WIDTH:16384
MAXIMUM_TEXTURE3D_HEIGHT:16384
MAXIMUM_TEXTURE3D_DEPTH:16384
MAXIMUM_TEXTURE2D_ARRAY_WIDTH:32768
MAXIMUM_TEXTURE2D_ARRAY_HEIGHT:32768
MAXIMUM_TEXTURE2D_ARRAY_NUMSLICES:2048
SURFACE_ALIGNMENT:512
CONCURRENT_KERNELS:1
ECC_ENABLED:0
PCI_BUS_ID:1
PCI_DEVICE_ID:0
TCC_DRIVER:0
MEMORY_CLOCK_RATE:5505000
GLOBAL_MEMORY_BUS_WIDTH:352
L2_CACHE_SIZE:2883584
MAX_THREADS_PER_MULTIPROCESSOR:2048
ASYNC_ENGINE_COUNT:2
UNIFIED_ADDRESSING:1
MAXIMUM_TEXTURE1D_LAYERED_WIDTH:32768
MAXIMUM_TEXTURE1D_LAYERED_LAYERS:2048
MAXIMUM_TEXTURE2D_GATHER_WIDTH:32768
MAXIMUM_TEXTURE2D_GATHER_HEIGHT:32768
MAXIMUM_TEXTURE3D_WIDTH_ALTERNATE:8192
MAXIMUM_TEXTURE3D_HEIGHT_ALTERNATE:8192
MAXIMUM_TEXTURE3D_DEPTH_ALTERNATE:32768
PCI_DOMAIN_ID:0
TEXTURE_PITCH_ALIGNMENT:32
MAXIMUM_TEXTURECUBEMAP_WIDTH:32768
MAXIMUM_TEXTURECUBEMAP_LAYERED_WIDTH:32768
MAXIMUM_TEXTURECUBEMAP_LAYERED_LAYERS:2046
MAXIMUM_SURFACE1D_WIDTH:32768
MAXIMUM_SURFACE2D_WIDTH:131072
MAXIMUM_SURFACE2D_HEIGHT:65536
MAXIMUM_SURFACE3D_WIDTH:16384
MAXIMUM_SURFACE3D_HEIGHT:16384
MAXIMUM_SURFACE3D_DEPTH:16384
MAXIMUM_SURFACE1D_LAYERED_WIDTH:32768
MAXIMUM_SURFACE1D_LAYERED_LAYERS:2048
MAXIMUM_SURFACE2D_LAYERED_WIDTH:32768
MAXIMUM_SURFACE2D_LAYERED_HEIGHT:32768
MAXIMUM_SURFACE2D_LAYERED_LAYERS:2048
MAXIMUM_SURFACECUBEMAP_WIDTH:32768
MAXIMUM_SURFACECUBEMAP_LAYERED_WIDTH:32768
MAXIMUM_SURFACECUBEMAP_LAYERED_LAYERS:2046
MAXIMUM_TEXTURE1D_LINEAR_WIDTH:134217728
MAXIMUM_TEXTURE2D_LINEAR_WIDTH:131072
MAXIMUM_TEXTURE2D_LINEAR_HEIGHT:65000
MAXIMUM_TEXTURE2D_LINEAR_PITCH:2097120
MAXIMUM_TEXTURE2D_MIPMAPPED_WIDTH:32768
MAXIMUM_TEXTURE2D_MIPMAPPED_HEIGHT:32768
COMPUTE_CAPABILITY_MAJOR:6
COMPUTE_CAPABILITY_MINOR:1
MAXIMUM_TEXTURE1D_MIPMAPPED_WIDTH:16384
STREAM_PRIORITIES_SUPPORTED:1
GLOBAL_L1_CACHE_SUPPORTED:1
LOCAL_L1_CACHE_SUPPORTED:1
MAX_SHARED_MEMORY_PER_MULTIPROCESSOR:98304
MAX_REGISTERS_PER_MULTIPROCESSOR:65536
MANAGED_MEMORY:1
MULTI_GPU_BOARD:0
MULTI_GPU_BOARD_GROUP_ID:0

##############################################

##############################################
#########01 PyTorch GPU support test.ipynb#########

__Python VERSION:’, ‘2.7.12 (default, Nov 19 2016, 06:48:10) \n[GCC 5.4.0 20160609]
__pyTorch VERSION:’, ‘0.1.12_2
__CUDA VERSION
nvcc: NVIDIA ® Cuda compiler driver
Copyright © 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176
__CUDNN VERSION:’, 6021
__Number CUDA Devices:’, 1L
Active CUDA Device: GPU’, 0L
Available devices ', 1L
Current cuda device ', 0L

##############################################
1 device(s) found.
(0, ‘GeForce GTX 1080 Ti’)

##############################################
True
<class ‘pycuda.gpuarray.GPUArray’>
##############################################

@bmkor please make sure you install the CUDA8 version of pytorch from the version selector. if you install version of 7.5 then you will have to wait a minute for startup.

@smth Thanks. Should I down my cuda from 9 to 8 in this case? I am not sure if my cuda is 9 in fact. Execute command nvcc —version gives:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176

@bmkor cuda >=8 should be fine. So if you have cuda9, then you dont need to downgrade.

1 Like

i install pytorch through anaconda.

I see a similar issue when running mytensor.to(torch.device('cuda'))

here’s the output to several calls to nvidia-smi, notice how the memory usage goes up until ~340MiB and then the call returns

Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:13 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   43C    P8    N/A /  N/A |      0MiB /   983MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:15 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   44C    P0    N/A /  N/A |     33MiB /   983MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15774      C   ...ymology/.virtualenvs/pytorch/bin/python    22MiB |
+-----------------------------------------------------------------------------+

Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:16 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   44C    P0    N/A /  N/A |     94MiB /   983MiB |     15%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15774      C   ...ymology/.virtualenvs/pytorch/bin/python    83MiB |
+-----------------------------------------------------------------------------+

Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:17 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   44C    P0    N/A /  N/A |    140MiB /   983MiB |      7%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15774      C   ...ymology/.virtualenvs/pytorch/bin/python   130MiB |
+-----------------------------------------------------------------------------+

Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:18 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   44C    P0    N/A /  N/A |    173MiB /   983MiB |      5%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15774      C   ...ymology/.virtualenvs/pytorch/bin/python   162MiB |
+-----------------------------------------------------------------------------+

Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:19 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   44C    P0    N/A /  N/A |    216MiB /   983MiB |      5%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15774      C   ...ymology/.virtualenvs/pytorch/bin/python   205MiB |
+-----------------------------------------------------------------------------+

Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:20 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   45C    P0    N/A /  N/A |    249MiB /   983MiB |      5%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15774      C   ...ymology/.virtualenvs/pytorch/bin/python   238MiB |
+-----------------------------------------------------------------------------+

Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:21 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   45C    P0    N/A /  N/A |    292MiB /   983MiB |      3%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15774      C   ...ymology/.virtualenvs/pytorch/bin/python   281MiB |
+-----------------------------------------------------------------------------+

Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:23 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   45C    P0    N/A /  N/A |    333MiB /   983MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15774      C   ...ymology/.virtualenvs/pytorch/bin/python   322MiB |
+-----------------------------------------------------------------------------+
^[[A                                                                                                                                                                                                              

Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:25 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   45C    P0    N/A /  N/A |    353MiB /   983MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15774      C   ...ymology/.virtualenvs/pytorch/bin/python   342MiB |
+-----------------------------------------------------------------------------+

I am using Driver Version: 515.43.04 with CUDA Version: 11.7 and with torch 1.12.1 on ubuntu Ubuntu 18.04.6 LTS.

I am facing the same issue. For me, if I reboot the system, it will work only once after that. The second time I try, it does not work.

1 Like