bmkor
(Benjamin)
November 8, 2017, 11:27am
1
Hi guys. I am a newbie in pytorch
.
My Ubuntu 16.04.02 LTS got python2.7 and cuda 9.0 installed (torch.cuda.is_available()
returns True
). However, when I tried run the following, my python just got frozen and I can only type ctrl+z
to force exit.
import torch
torch.randn(10).cuda()
Then freeze until ctrl+z
.
Did I do something wrong? Anyone could help? Thanks a lot.
Hi, i see similar issue during use CUDA.
And I try to check GPU memory, it will goes up until ~229MB. Python then can work normally.
I am not sure why. If you find a way to fix it, please tell me. thanks
lgvaz
(Lucas Vazquez)
November 8, 2017, 2:12pm
3
I’m facing the same issue.
I’m running on jupyter notebook, the first call to torch.randn(10).cuda()
takes a long loooong time to complete (~7 min), but subsequent calls behave normally.
I’m posting a screenshoot of the stack when interrupting the kernel.
1 Like
richard
(Richard Zou)
November 8, 2017, 2:56pm
4
What is the output of your nvidia-smi when this is happening?
Could you try building PyTorch from source and seeing if the long cuda() calls still occur?
lgvaz
(Lucas Vazquez)
November 8, 2017, 3:58pm
5
This is the output of nvidia-smi (fastai/bin/python is the process related to the notebook):
I will try to build from source.
lgvaz
(Lucas Vazquez)
November 8, 2017, 8:58pm
6
I couldn’t manage to build from source, first I was getting this error, I followed the workaround but then got an error regarding cuDNN version, my cuDNN installation is a mess because of tensorflow, so I didn’t wanted to mess with that afraid that it could mess my tensorflow installation. So I switched to docker, and the first run of torch.randn(10).cuda()
is taking ~2 sec, which I think is expected.
smth
November 9, 2017, 1:35am
7
@lgvaz you have a 1070 GPU, but you must’ve installed pytorch for CUDA 7.5. What it is doing is recompiling pytorch for your GPU (cuda8 supported GPU).
If you reinstall the CUDA8 version of pytorch, the freezing wont happen. Select the right option in our version selector on http://pytorch.org
2 Likes
Hi smth,
i also see the same issue even with CUDA8.
richard
(Richard Zou)
November 9, 2017, 3:11pm
9
@Spencer_ML It’s possible that there’s a mismatch between the CUDA version your pytorch was compiled with, and the CUDA version you’re running your computer on. How did you install pytorch?
bmkor
(Benjamin)
November 11, 2017, 7:13am
10
Thanks all for the discussion.
Have successfully tried cuda() but on the first run of .cuda(), I found that I have to wait for 5 to 10 minutes. Is it normal or should I do some config first?
Can you run the following and report the output?
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Deep Learning Bootcamp November 2017, GPU Computing for Data Scientists\n",
"\n",
"<img src=\"../images/bcamp.png\" align=\"center\">\n",
"\n",
"## Using CUDA, Jupyter, PyCUDA and PyTorch\n",
"\n",
"### 01 PyCUDA verify CUDA 8.0\n",
"\n",
"Web: https://www.meetup.com/Tel-Aviv-Deep-Learning-Bootcamp/events/241762893/\n",
"\n",
This file has been truncated. show original
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"% reset -f\n",
"from __future__ import print_function\n",
"from __future__ import division\n",
"import math\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"%matplotlib inline\n",
"# ! pip install torchvision"
]
},
This file has been truncated. show original
bmkor
(Benjamin)
November 12, 2017, 2:27am
12
@QuantScientist Sure. Here are my outputs:
##############################################
##########01 PyCUDA verify CUDA 8.0.ipynb########
1 device(s) found.
Device #0: GeForce GTX 1080 Ti
Compute Capability: 6.1
Total Memory: 11440512 KB
ASYNC_ENGINE_COUNT: 2
CAN_MAP_HOST_MEMORY: 1
CLOCK_RATE: 1582000
COMPUTE_CAPABILITY_MAJOR: 6
COMPUTE_CAPABILITY_MINOR: 1
COMPUTE_MODE: DEFAULT
CONCURRENT_KERNELS: 1
ECC_ENABLED: 0
GLOBAL_L1_CACHE_SUPPORTED: 1
GLOBAL_MEMORY_BUS_WIDTH: 352
GPU_OVERLAP: 1
INTEGRATED: 0
KERNEL_EXEC_TIMEOUT: 1
L2_CACHE_SIZE: 2883584
LOCAL_L1_CACHE_SUPPORTED: 1
MANAGED_MEMORY: 1
MAXIMUM_SURFACE1D_LAYERED_LAYERS: 2048
MAXIMUM_SURFACE1D_LAYERED_WIDTH: 32768
MAXIMUM_SURFACE1D_WIDTH: 32768
MAXIMUM_SURFACE2D_HEIGHT: 65536
MAXIMUM_SURFACE2D_LAYERED_HEIGHT: 32768
MAXIMUM_SURFACE2D_LAYERED_LAYERS: 2048
MAXIMUM_SURFACE2D_LAYERED_WIDTH: 32768
MAXIMUM_SURFACE2D_WIDTH: 131072
MAXIMUM_SURFACE3D_DEPTH: 16384
MAXIMUM_SURFACE3D_HEIGHT: 16384
MAXIMUM_SURFACE3D_WIDTH: 16384
MAXIMUM_SURFACECUBEMAP_LAYERED_LAYERS: 2046
MAXIMUM_SURFACECUBEMAP_LAYERED_WIDTH: 32768
MAXIMUM_SURFACECUBEMAP_WIDTH: 32768
MAXIMUM_TEXTURE1D_LAYERED_LAYERS: 2048
MAXIMUM_TEXTURE1D_LAYERED_WIDTH: 32768
MAXIMUM_TEXTURE1D_LINEAR_WIDTH: 134217728
MAXIMUM_TEXTURE1D_MIPMAPPED_WIDTH: 16384
MAXIMUM_TEXTURE1D_WIDTH: 131072
MAXIMUM_TEXTURE2D_ARRAY_HEIGHT: 32768
MAXIMUM_TEXTURE2D_ARRAY_NUMSLICES: 2048
MAXIMUM_TEXTURE2D_ARRAY_WIDTH: 32768
MAXIMUM_TEXTURE2D_GATHER_HEIGHT: 32768
MAXIMUM_TEXTURE2D_GATHER_WIDTH: 32768
MAXIMUM_TEXTURE2D_HEIGHT: 65536
MAXIMUM_TEXTURE2D_LINEAR_HEIGHT: 65000
MAXIMUM_TEXTURE2D_LINEAR_PITCH: 2097120
MAXIMUM_TEXTURE2D_LINEAR_WIDTH: 131072
MAXIMUM_TEXTURE2D_MIPMAPPED_HEIGHT: 32768
MAXIMUM_TEXTURE2D_MIPMAPPED_WIDTH: 32768
MAXIMUM_TEXTURE2D_WIDTH: 131072
MAXIMUM_TEXTURE3D_DEPTH: 16384
MAXIMUM_TEXTURE3D_DEPTH_ALTERNATE: 32768
MAXIMUM_TEXTURE3D_HEIGHT: 16384
MAXIMUM_TEXTURE3D_HEIGHT_ALTERNATE: 8192
MAXIMUM_TEXTURE3D_WIDTH: 16384
MAXIMUM_TEXTURE3D_WIDTH_ALTERNATE: 8192
MAXIMUM_TEXTURECUBEMAP_LAYERED_LAYERS: 2046
MAXIMUM_TEXTURECUBEMAP_LAYERED_WIDTH: 32768
MAXIMUM_TEXTURECUBEMAP_WIDTH: 32768
MAX_BLOCK_DIM_X: 1024
MAX_BLOCK_DIM_Y: 1024
MAX_BLOCK_DIM_Z: 64
MAX_GRID_DIM_X: 2147483647
MAX_GRID_DIM_Y: 65535
MAX_GRID_DIM_Z: 65535
MAX_PITCH: 2147483647
MAX_REGISTERS_PER_BLOCK: 65536
MAX_REGISTERS_PER_MULTIPROCESSOR: 65536
MAX_SHARED_MEMORY_PER_BLOCK: 49152
MAX_SHARED_MEMORY_PER_MULTIPROCESSOR: 98304
MAX_THREADS_PER_BLOCK: 1024
MAX_THREADS_PER_MULTIPROCESSOR: 2048
MEMORY_CLOCK_RATE: 5505000
MULTIPROCESSOR_COUNT: 28
MULTI_GPU_BOARD: 0
MULTI_GPU_BOARD_GROUP_ID: 0
PCI_BUS_ID: 1
PCI_DEVICE_ID: 0
PCI_DOMAIN_ID: 0
STREAM_PRIORITIES_SUPPORTED: 1
SURFACE_ALIGNMENT: 512
TCC_DRIVER: 0
TEXTURE_ALIGNMENT: 512
TEXTURE_PITCH_ALIGNMENT: 32
TOTAL_CONSTANT_MEMORY: 65536
UNIFIED_ADDRESSING: 1
WARP_SIZE: 32
##############################################
Global memory occupancy:98.000000% free
===Attributes for device 0
MAX_THREADS_PER_BLOCK:1024
MAX_BLOCK_DIM_X:1024
MAX_BLOCK_DIM_Y:1024
MAX_BLOCK_DIM_Z:64
MAX_GRID_DIM_X:2147483647
MAX_GRID_DIM_Y:65535
MAX_GRID_DIM_Z:65535
MAX_SHARED_MEMORY_PER_BLOCK:49152
TOTAL_CONSTANT_MEMORY:65536
WARP_SIZE:32
MAX_PITCH:2147483647
MAX_REGISTERS_PER_BLOCK:65536
CLOCK_RATE:1582000
TEXTURE_ALIGNMENT:512
GPU_OVERLAP:1
MULTIPROCESSOR_COUNT:28
KERNEL_EXEC_TIMEOUT:1
INTEGRATED:0
CAN_MAP_HOST_MEMORY:1
COMPUTE_MODE:DEFAULT
MAXIMUM_TEXTURE1D_WIDTH:131072
MAXIMUM_TEXTURE2D_WIDTH:131072
MAXIMUM_TEXTURE2D_HEIGHT:65536
MAXIMUM_TEXTURE3D_WIDTH:16384
MAXIMUM_TEXTURE3D_HEIGHT:16384
MAXIMUM_TEXTURE3D_DEPTH:16384
MAXIMUM_TEXTURE2D_ARRAY_WIDTH:32768
MAXIMUM_TEXTURE2D_ARRAY_HEIGHT:32768
MAXIMUM_TEXTURE2D_ARRAY_NUMSLICES:2048
SURFACE_ALIGNMENT:512
CONCURRENT_KERNELS:1
ECC_ENABLED:0
PCI_BUS_ID:1
PCI_DEVICE_ID:0
TCC_DRIVER:0
MEMORY_CLOCK_RATE:5505000
GLOBAL_MEMORY_BUS_WIDTH:352
L2_CACHE_SIZE:2883584
MAX_THREADS_PER_MULTIPROCESSOR:2048
ASYNC_ENGINE_COUNT:2
UNIFIED_ADDRESSING:1
MAXIMUM_TEXTURE1D_LAYERED_WIDTH:32768
MAXIMUM_TEXTURE1D_LAYERED_LAYERS:2048
MAXIMUM_TEXTURE2D_GATHER_WIDTH:32768
MAXIMUM_TEXTURE2D_GATHER_HEIGHT:32768
MAXIMUM_TEXTURE3D_WIDTH_ALTERNATE:8192
MAXIMUM_TEXTURE3D_HEIGHT_ALTERNATE:8192
MAXIMUM_TEXTURE3D_DEPTH_ALTERNATE:32768
PCI_DOMAIN_ID:0
TEXTURE_PITCH_ALIGNMENT:32
MAXIMUM_TEXTURECUBEMAP_WIDTH:32768
MAXIMUM_TEXTURECUBEMAP_LAYERED_WIDTH:32768
MAXIMUM_TEXTURECUBEMAP_LAYERED_LAYERS:2046
MAXIMUM_SURFACE1D_WIDTH:32768
MAXIMUM_SURFACE2D_WIDTH:131072
MAXIMUM_SURFACE2D_HEIGHT:65536
MAXIMUM_SURFACE3D_WIDTH:16384
MAXIMUM_SURFACE3D_HEIGHT:16384
MAXIMUM_SURFACE3D_DEPTH:16384
MAXIMUM_SURFACE1D_LAYERED_WIDTH:32768
MAXIMUM_SURFACE1D_LAYERED_LAYERS:2048
MAXIMUM_SURFACE2D_LAYERED_WIDTH:32768
MAXIMUM_SURFACE2D_LAYERED_HEIGHT:32768
MAXIMUM_SURFACE2D_LAYERED_LAYERS:2048
MAXIMUM_SURFACECUBEMAP_WIDTH:32768
MAXIMUM_SURFACECUBEMAP_LAYERED_WIDTH:32768
MAXIMUM_SURFACECUBEMAP_LAYERED_LAYERS:2046
MAXIMUM_TEXTURE1D_LINEAR_WIDTH:134217728
MAXIMUM_TEXTURE2D_LINEAR_WIDTH:131072
MAXIMUM_TEXTURE2D_LINEAR_HEIGHT:65000
MAXIMUM_TEXTURE2D_LINEAR_PITCH:2097120
MAXIMUM_TEXTURE2D_MIPMAPPED_WIDTH:32768
MAXIMUM_TEXTURE2D_MIPMAPPED_HEIGHT:32768
COMPUTE_CAPABILITY_MAJOR:6
COMPUTE_CAPABILITY_MINOR:1
MAXIMUM_TEXTURE1D_MIPMAPPED_WIDTH:16384
STREAM_PRIORITIES_SUPPORTED:1
GLOBAL_L1_CACHE_SUPPORTED:1
LOCAL_L1_CACHE_SUPPORTED:1
MAX_SHARED_MEMORY_PER_MULTIPROCESSOR:98304
MAX_REGISTERS_PER_MULTIPROCESSOR:65536
MANAGED_MEMORY:1
MULTI_GPU_BOARD:0
MULTI_GPU_BOARD_GROUP_ID:0
##############################################
##############################################
#########01 PyTorch GPU support test.ipynb#########
__Python VERSION:’, ‘2.7.12 (default, Nov 19 2016, 06:48:10) \n[GCC 5.4.0 20160609]
__pyTorch VERSION:’, ‘0.1.12_2
__CUDA VERSION
nvcc: NVIDIA ® Cuda compiler driver
Copyright © 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176
__CUDNN VERSION:’, 6021
__Number CUDA Devices:’, 1L
Active CUDA Device: GPU’, 0L
Available devices ', 1L
Current cuda device ', 0L
##############################################
1 device(s) found.
(0, ‘GeForce GTX 1080 Ti’)
##############################################
True
<class ‘pycuda.gpuarray.GPUArray’>
##############################################
smth
November 12, 2017, 7:13pm
13
@bmkor please make sure you install the CUDA8 version of pytorch from the version selector. if you install version of 7.5 then you will have to wait a minute for startup.
bmkor
(Benjamin)
November 15, 2017, 12:51pm
14
@smth Thanks. Should I down my cuda from 9 to 8 in this case? I am not sure if my cuda is 9 in fact. Execute command nvcc —version
gives:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176
smth
November 15, 2017, 2:27pm
15
@bmkor cuda >=8 should be fine. So if you have cuda9, then you dont need to downgrade.
1 Like
i install pytorch through anaconda.
I see a similar issue when running mytensor.to(torch.device('cuda'))
here’s the output to several calls to nvidia-smi, notice how the memory usage goes up until ~340MiB and then the call returns
Shiphero-API on feature/tomas/OP-11-mergeable-order-large-accounts [$?]
➜ nvidia-smi
Thu Oct 18 19:45:13 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57 Driver Version: 410.57 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce 940M Off | 00000000:04:00.0 Off | N/A |
| N/A 43C P8 N/A / N/A | 0MiB / 983MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
Shiphero-API on feature/tomas/OP-11-mergeable-order-large-accounts [$?]
➜ nvidia-smi
Thu Oct 18 19:45:15 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57 Driver Version: 410.57 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce 940M Off | 00000000:04:00.0 Off | N/A |
| N/A 44C P0 N/A / N/A | 33MiB / 983MiB | 1% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 15774 C ...ymology/.virtualenvs/pytorch/bin/python 22MiB |
+-----------------------------------------------------------------------------+
Shiphero-API on feature/tomas/OP-11-mergeable-order-large-accounts [$?]
➜ nvidia-smi
Thu Oct 18 19:45:16 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57 Driver Version: 410.57 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce 940M Off | 00000000:04:00.0 Off | N/A |
| N/A 44C P0 N/A / N/A | 94MiB / 983MiB | 15% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 15774 C ...ymology/.virtualenvs/pytorch/bin/python 83MiB |
+-----------------------------------------------------------------------------+
Shiphero-API on feature/tomas/OP-11-mergeable-order-large-accounts [$?]
➜ nvidia-smi
Thu Oct 18 19:45:17 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57 Driver Version: 410.57 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce 940M Off | 00000000:04:00.0 Off | N/A |
| N/A 44C P0 N/A / N/A | 140MiB / 983MiB | 7% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 15774 C ...ymology/.virtualenvs/pytorch/bin/python 130MiB |
+-----------------------------------------------------------------------------+
Shiphero-API on feature/tomas/OP-11-mergeable-order-large-accounts [$?]
➜ nvidia-smi
Thu Oct 18 19:45:18 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57 Driver Version: 410.57 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce 940M Off | 00000000:04:00.0 Off | N/A |
| N/A 44C P0 N/A / N/A | 173MiB / 983MiB | 5% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 15774 C ...ymology/.virtualenvs/pytorch/bin/python 162MiB |
+-----------------------------------------------------------------------------+
Shiphero-API on feature/tomas/OP-11-mergeable-order-large-accounts [$?]
➜ nvidia-smi
Thu Oct 18 19:45:19 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57 Driver Version: 410.57 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce 940M Off | 00000000:04:00.0 Off | N/A |
| N/A 44C P0 N/A / N/A | 216MiB / 983MiB | 5% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 15774 C ...ymology/.virtualenvs/pytorch/bin/python 205MiB |
+-----------------------------------------------------------------------------+
Shiphero-API on feature/tomas/OP-11-mergeable-order-large-accounts [$?]
➜ nvidia-smi
Thu Oct 18 19:45:20 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57 Driver Version: 410.57 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce 940M Off | 00000000:04:00.0 Off | N/A |
| N/A 45C P0 N/A / N/A | 249MiB / 983MiB | 5% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 15774 C ...ymology/.virtualenvs/pytorch/bin/python 238MiB |
+-----------------------------------------------------------------------------+
Shiphero-API on feature/tomas/OP-11-mergeable-order-large-accounts [$?]
➜ nvidia-smi
Thu Oct 18 19:45:21 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57 Driver Version: 410.57 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce 940M Off | 00000000:04:00.0 Off | N/A |
| N/A 45C P0 N/A / N/A | 292MiB / 983MiB | 3% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 15774 C ...ymology/.virtualenvs/pytorch/bin/python 281MiB |
+-----------------------------------------------------------------------------+
Shiphero-API on feature/tomas/OP-11-mergeable-order-large-accounts [$?]
➜ nvidia-smi
Thu Oct 18 19:45:23 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57 Driver Version: 410.57 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce 940M Off | 00000000:04:00.0 Off | N/A |
| N/A 45C P0 N/A / N/A | 333MiB / 983MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 15774 C ...ymology/.virtualenvs/pytorch/bin/python 322MiB |
+-----------------------------------------------------------------------------+
^[[A
Shiphero-API on feature/tomas/OP-11-mergeable-order-large-accounts [$?]
➜ nvidia-smi
Thu Oct 18 19:45:25 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57 Driver Version: 410.57 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce 940M Off | 00000000:04:00.0 Off | N/A |
| N/A 45C P0 N/A / N/A | 353MiB / 983MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 15774 C ...ymology/.virtualenvs/pytorch/bin/python 342MiB |
+-----------------------------------------------------------------------------+
I am using Driver Version: 515.43.04
with CUDA Version: 11.7
and with torch 1.12.1
on ubuntu Ubuntu 18.04.6 LTS
.
I am facing the same issue. For me, if I reboot the system, it will work only once after that. The second time I try, it does not work.
1 Like