Here is the output:
during the first two runs it looks like this:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01 Driver Version: 440.33.01 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla V100-PCIE... Off | 00000000:18:00.0 Off | 0 |
| N/A 44C P0 38W / 250W | 2857MiB / 16160MiB | 0% E. Process |
+-------------------------------+----------------------+----------------------+
| 1 Tesla V100-PCIE... Off | 00000000:3B:00.0 Off | 0 |
| N/A 43C P0 27W / 250W | 12MiB / 16160MiB | 0% E. Process |
+-------------------------------+----------------------+----------------------+
| 2 Tesla V100-PCIE... Off | 00000000:86:00.0 Off | 0 |
| N/A 41C P0 27W / 250W | 12MiB / 16160MiB | 0% E. Process |
+-------------------------------+----------------------+----------------------+
| 3 Tesla V100-PCIE... Off | 00000000:AF:00.0 Off | 0 |
| N/A 41C P0 28W / 250W | 12MiB / 16160MiB | 0% E. Process |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 284149 C python 2845MiB |
+-----------------------------------------------------------------------------+
During the last run:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01 Driver Version: 440.33.01 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla V100-PCIE... Off | 00000000:18:00.0 Off | 0 |
| N/A 46C P0 43W / 250W | 14527MiB / 16160MiB | 0% E. Process |
+-------------------------------+----------------------+----------------------+
| 1 Tesla V100-PCIE... Off | 00000000:3B:00.0 Off | 0 |
| N/A 42C P0 27W / 250W | 12MiB / 16160MiB | 0% E. Process |
+-------------------------------+----------------------+----------------------+
| 2 Tesla V100-PCIE... Off | 00000000:86:00.0 Off | 0 |
| N/A 41C P0 27W / 250W | 12MiB / 16160MiB | 0% E. Process |
+-------------------------------+----------------------+----------------------+
| 3 Tesla V100-PCIE... Off | 00000000:AF:00.0 Off | 0 |
| N/A 41C P0 28W / 250W | 12MiB / 16160MiB | 0% E. Process |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 284149 C python 14515MiB |
+-----------------------------------------------------------------------------+