Laptop shuts down while trainining

1

My laptop automatically shuts without warning (sometimes) during training.

Machine config - Lenovo y540, RTX 2060, Ubuntu 18.04, PyTorch 1.4. I tried training a simple binary image classification model (4 conv layers with batchnorm and Dropout). The model trained for 20 epochs (batch size = 8) and then my laptop shut down.

Output of nvidia-smi:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.50       Driver Version: 430.50       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 2060    Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   47C    P8     3W /  N/A |     10MiB /  5934MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Following is the log file before the system crashed I think. I found it in - cat /var/log/kern.log.

Mar 10 17:05:01 maverick kernel: [    9.279289] audit: type=1400 audit(1583840101.525:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=837 comm="apparmor_parser"
Mar 10 17:05:01 maverick kernel: [    9.280042] audit: type=1400 audit(1583840101.529:11): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/sbin/dhclient" pid=828 comm="apparmor_parser"
Mar 10 17:05:01 maverick kernel: [    9.325087] intel_rapl_common: Found RAPL domain package
Mar 10 17:05:01 maverick kernel: [    9.325092] intel_rapl_common: Found RAPL domain core
Mar 10 17:05:01 maverick kernel: [    9.325096] intel_rapl_common: Found RAPL domain uncore
Mar 10 17:05:01 maverick kernel: [    9.325100] intel_rapl_common: Found RAPL domain dram
Mar 10 17:05:01 maverick kernel: [    9.355748] input: HDA Intel PCH Mic as /devices/pci0000:00/0000:00:1f.3/sound/card0/input13
Mar 10 17:05:01 maverick kernel: [    9.355987] input: HDA Intel PCH Headphone as /devices/pci0000:00/0000:00:1f.3/sound/card0/input14
Mar 10 17:05:01 maverick kernel: [    9.356199] input: HDA Intel PCH HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input15
Mar 10 17:05:01 maverick kernel: [    9.356895] input: HDA Intel PCH HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input16
Mar 10 17:05:01 maverick kernel: [    9.357074] input: HDA Intel PCH HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input17
Mar 10 17:05:01 maverick kernel: [    9.357296] input: HDA Intel PCH HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input18
Mar 10 17:05:01 maverick kernel: [    9.357497] input: HDA Intel PCH HDMI/DP,pcm=10 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input19
Mar 10 17:05:01 maverick kernel: [    9.432866] dw-apb-uart.2: ttyS4 at MMIO 0x8f802000 (irq = 20, base_baud = 115200) is a 16550A
Mar 10 17:05:01 maverick kernel: [    9.434397] iwlwifi 0000:00:14.3 wlp0s20f3: renamed from wlan0
Mar 10 17:05:01 maverick kernel: [    9.445610] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms  430.50  Thu Sep  5 22:39:50 CDT 2019
Mar 10 17:05:01 maverick kernel: [    9.575171] nvidia-uvm: Loaded the UVM driver in 8 mode, major device number 234
Mar 10 17:05:01 maverick kernel: [    9.623512] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
Mar 10 17:05:01 maverick kernel: [    9.623516] Bluetooth: BNEP filters: protocol multicast
Mar 10 17:05:01 maverick kernel: [    9.623525] Bluetooth: BNEP socket layer initialized
Mar 10 17:05:01 maverick kernel: [    9.664785] input: MSFT0001:01 06CB:CD5F Touchpad as /devices/pci0000:00/0000:00:15.1/i2c_designware.1/i2c-2/i2c-MSFT0001:01/0018:06CB:CD5F.0003/input/input24
Mar 10 17:05:01 maverick kernel: [    9.665154] hid-multitouch 0018:06CB:CD5F.0003: input,hidraw2: I2C HID v1.00 Mouse [MSFT0001:01 06CB:CD5F] on i2c-MSFT0001:01
Mar 10 17:05:01 maverick kernel: [    9.669632] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card1/input20
Mar 10 17:05:01 maverick kernel: [    9.669880] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card1/input21
Mar 10 17:05:01 maverick kernel: [    9.669932] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card1/input22
Mar 10 17:05:02 maverick kernel: [    9.767641] ACPI Warning: \_SB.PCI0.PEG0.PEGP._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20190703/nsarguments-66)
Mar 10 17:05:02 maverick kernel: [   10.035982] Generic Realtek PHY r8169-700:00: attached PHY driver [Generic Realtek PHY] (mii_bus:phy_addr=r8169-700:00, irq=IGNORE)
Mar 10 17:05:02 maverick kernel: [   10.149333] r8169 0000:07:00.0 enp7s0: Link is Down
Mar 10 17:05:02 maverick kernel: [   10.179246] iwlwifi 0000:00:14.3: Applying debug destination EXTERNAL_DRAM
Mar 10 17:05:02 maverick kernel: [   10.296096] iwlwifi 0000:00:14.3: Applying debug destination EXTERNAL_DRAM
Mar 10 17:05:02 maverick kernel: [   10.361833] iwlwifi 0000:00:14.3: FW already configured (0) - re-configuring
Mar 10 17:05:02 maverick kernel: [   10.374304] iwlwifi 0000:00:14.3: BIOS contains WGDS but no WRDS
Mar 10 17:05:02 maverick kernel: [   10.378535] Bluetooth: hci0: Waiting for firmware download to complete
Mar 10 17:05:02 maverick kernel: [   10.379322] Bluetooth: hci0: Firmware loaded in 1598306 usecs
Mar 10 17:05:02 maverick kernel: [   10.379451] Bluetooth: hci0: Waiting for device to boot
Mar 10 17:05:02 maverick kernel: [   10.392359] Bluetooth: hci0: Device booted in 12671 usecs
Mar 10 17:05:02 maverick kernel: [   10.395240] Bluetooth: hci0: Found Intel DDC parameters: intel/ibt-17-16-1.ddc
Mar 10 17:05:02 maverick kernel: [   10.398388] Bluetooth: hci0: Applying Intel DDC parameters completed
Mar 10 17:05:03 maverick kernel: [   11.148057] nvidia-uvm: Unloaded the UVM driver in 8 mode
Mar 10 17:05:03 maverick kernel: [   11.171826] nvidia-modeset: Unloading
Mar 10 17:05:03 maverick kernel: [   11.219065] nvidia-nvlink: Unregistered the Nvlink Core, major device number 237
Mar 10 17:05:04 maverick kernel: [   12.125832] nvidia-nvlink: Nvlink Core is being initialized, major device number 237
Mar 10 17:05:04 maverick kernel: [   12.127484] nvidia 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=none
Mar 10 17:05:04 maverick kernel: [   12.175644] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  430.50  Thu Sep  5 22:36:31 CDT 2019
Mar 10 17:05:05 maverick kernel: [   13.205291] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms  430.50  Thu Sep  5 22:39:50 CDT 2019
Mar 10 17:05:05 maverick kernel: [   13.250663] nvidia-uvm: Loaded the UVM driver in 8 mode, major device number 234
Mar 10 17:05:06 maverick kernel: [   13.986003] wlp0s20f3: authenticate with 58:c1:7a:1b:bd:d0
Mar 10 17:05:06 maverick kernel: [   13.994385] wlp0s20f3: send auth to 58:c1:7a:1b:bd:d0 (try 1/3)
Mar 10 17:05:06 maverick kernel: [   14.047103] iwlwifi 0000:00:14.3: Unhandled alg: 0x707
Mar 10 17:05:06 maverick kernel: [   14.063692] wlp0s20f3: authenticated
Mar 10 17:05:06 maverick kernel: [   14.068040] wlp0s20f3: associate with 58:c1:7a:1b:bd:d0 (try 1/3)
Mar 10 17:05:06 maverick kernel: [   14.097924] wlp0s20f3: RX AssocResp from 58:c1:7a:1b:bd:d0 (capab=0x431 status=0 aid=4)
Mar 10 17:05:06 maverick kernel: [   14.143288] iwlwifi 0000:00:14.3: Unhandled alg: 0x707
Mar 10 17:05:06 maverick kernel: [   14.177499] wlp0s20f3: associated
Mar 10 17:05:06 maverick kernel: [   14.296025] IPv6: ADDRCONF(NETDEV_CHANGE): wlp0s20f3: link becomes ready
Mar 10 17:05:08 maverick kernel: [   16.376337] bpfilter: Loaded bpfilter_umh pid 1511
Mar 10 17:05:18 maverick kernel: [   26.325876] Bluetooth: RFCOMM TTY layer initialized
Mar 10 17:05:18 maverick kernel: [   26.325884] Bluetooth: RFCOMM socket layer initialized
Mar 10 17:05:18 maverick kernel: [   26.325892] Bluetooth: RFCOMM ver 1.11
Mar 10 17:05:19 maverick kernel: [   27.169380] rfkill: input handler disabled
Mar 10 17:08:10 maverick kernel: [  198.039283] ucsi_ccg 0-0008: failed to reset PPM!
Mar 10 17:08:10 maverick kernel: [  198.039292] ucsi_ccg 0-0008: PPM init failed (-110)
Mar 10 17:10:11 maverick kernel: [  319.690728] mce: CPU11: Core temperature above threshold, cpu clock throttled (total events = 75)
Mar 10 17:10:11 maverick kernel: [  319.690729] mce: CPU5: Core temperature above threshold, cpu clock throttled (total events = 75)
Mar 10 17:10:11 maverick kernel: [  319.690730] mce: CPU11: Package temperature above threshold, cpu clock throttled (total events = 290)
Mar 10 17:10:11 maverick kernel: [  319.690730] mce: CPU5: Package temperature above threshold, cpu clock throttled (total events = 290)
Mar 10 17:10:11 maverick kernel: [  319.690772] mce: CPU6: Package temperature above threshold, cpu clock throttled (total events = 290)
Mar 10 17:10:11 maverick kernel: [  319.690773] mce: CPU1: Package temperature above threshold, cpu clock throttled (total events = 290)
Mar 10 17:10:11 maverick kernel: [  319.690774] mce: CPU0: Package temperature above threshold, cpu clock throttled (total events = 290)
Mar 10 17:10:11 maverick kernel: [  319.690775] mce: CPU3: Package temperature above threshold, cpu clock throttled (total events = 290)
Mar 10 17:10:11 maverick kernel: [  319.690776] mce: CPU7: Package temperature above threshold, cpu clock throttled (total events = 290)
Mar 10 17:10:11 maverick kernel: [  319.690777] mce: CPU9: Package temperature above threshold, cpu clock throttled (total events = 290)
Mar 10 17:10:11 maverick kernel: [  319.690778] mce: CPU4: Package temperature above threshold, cpu clock throttled (total events = 290)
Mar 10 17:10:11 maverick kernel: [  319.690779] mce: CPU2: Package temperature above threshold, cpu clock throttled (total events = 290)
Mar 10 17:10:11 maverick kernel: [  319.690780] mce: CPU10: Package temperature above threshold, cpu clock throttled (total events = 290)
Mar 10 17:10:11 maverick kernel: [  319.690781] mce: CPU8: Package temperature above threshold, cpu clock throttled (total events = 290)
Mar 10 17:10:11 maverick kernel: [  319.691710] mce: CPU5: Core temperature/speed normal
Mar 10 17:10:11 maverick kernel: [  319.691713] mce: CPU11: Core temperature/speed normal
Mar 10 17:10:11 maverick kernel: [  319.691716] mce: CPU11: Package temperature/speed normal
Mar 10 17:10:11 maverick kernel: [  319.691717] mce: CPU5: Package temperature/speed normal
Mar 10 17:10:11 maverick kernel: [  319.691777] mce: CPU0: Package temperature/speed normal
Mar 10 17:10:11 maverick kernel: [  319.691781] mce: CPU7: Package temperature/speed normal
Mar 10 17:10:11 maverick kernel: [  319.691783] mce: CPU6: Package temperature/speed normal
Mar 10 17:10:11 maverick kernel: [  319.691787] mce: CPU2: Package temperature/speed normal
Mar 10 17:10:11 maverick kernel: [  319.691790] mce: CPU1: Package temperature/speed normal
Mar 10 17:10:11 maverick kernel: [  319.691793] mce: CPU8: Package temperature/speed normal
Mar 10 17:10:11 maverick kernel: [  319.691798] mce: CPU10: Package temperature/speed normal
Mar 10 17:10:11 maverick kernel: [  319.691800] mce: CPU4: Package temperature/speed normal
Mar 10 17:10:11 maverick kernel: [  319.691804] mce: CPU3: Package temperature/speed normal
Mar 10 17:10:11 maverick kernel: [  319.691807] mce: CPU9: Package temperature/speed normal
Mar 10 17:13:35 maverick kernel: [  523.048575] wlp0s20f3: authenticate with 58:c1:7a:1b:bd:d0
Mar 10 17:13:35 maverick kernel: [  523.055288] wlp0s20f3: send auth to 58:c1:7a:1b:bd:d0 (try 1/3)
Mar 10 17:13:35 maverick kernel: [  523.097819] wlp0s20f3: authenticated
Mar 10 17:13:35 maverick kernel: [  523.099819] wlp0s20f3: associate with 58:c1:7a:1b:bd:d0 (try 1/3)
Mar 10 17:13:35 maverick kernel: [  523.107873] wlp0s20f3: RX AssocResp from 58:c1:7a:1b:bd:d0 (capab=0x431 status=0 aid=1)
Mar 10 17:13:35 maverick kernel: [  523.109523] iwlwifi 0000:00:14.3: Unhandled alg: 0x707
Mar 10 17:13:35 maverick kernel: [  523.110798] wlp0s20f3: associated
Mar 10 17:13:35 maverick kernel: [  523.119975] IPv6: ADDRCONF(NETDEV_CHANGE): wlp0s20f3: link becomes ready

How can I stop this from happening again ie. stop pytorch training and not crash my system?

2

Hello Mr. Air!

Your laptop is overheating and shutting itself down to prevent permanent
damage.

When your cpu and/or gpu is running full tilt – and pytorch tries to make
it run as fast as it can – that creates a lot of heat. (So pytorch is doing
its job here!)

Some things you can try:

Make sure your laptop has good ventilation. Don’t have it sitting on a
fluffy blanket or let your cat sleep blocking the cooling ports.

Make sure your cooling ports aren’t blocked or clogged up with lint.

Make sure your cooling fans are working properly. (Lenovo made a
series of laptops where the cooling fans would fail / get noisy as they
got older. This is something of a known issue.)

Try running your laptop on a cooling rack (those things you put your
cookies on after you take them out of the oven.)

Aim a fan or air conditioner at your laptop.

Open up the case of your laptop for better air flow – but, with a laptop,
I probably wouldn’t do this.

Squirt “compressed air” into your cooling ports – it gets quite cold as
it expands.

In your os power-management settings or bios you may have an option
to forcibly slow your laptop down. This will slow your training down,
but it will generate less heat.

Train on the cpu instead of the gpu (or vice versa). One or the other
may have better cooling versus heat generation. But training on the
cpu will very likely run more slowly.

If worst comes to worst, and none of the above work, you may have to
resort to training your model in stages. Run for a short enough amount
of time that your laptop doesn’t overheat / shut down, checkpoint your
model, let your laptop idle long enough to cool off some, reload your
checkpoint and train some more for a short period of time. Repeat.

Good luck.

K. Frank

2 Likes
3

Thank you for the detailed answer @KFrank! :smiley:

So, it’s all because of overheating. GPU memory or Power input has no role to play here?

I’ll look for a cooling pad now. Is there anything else I can do from PyTorch to alleviate this?

4

I don’t think you can do anything from PyTorch, but you should underclock your GPU with nvidia-smi

1 Like
5

Hi Mr. Air!

Almost certainly. Your log file was complaining about your CPUs
overheating right before the shutdown.

I would think that you would get an explicit GPU out-of-memory
error if GPU memory were the problem. Certainly not a hard
system shutdown.

There are power supplies (and batteries) that will shut themselves
down abruptly if they start to overheat, but, again, your log did say
that your CPUs were getting hot.

You could try running on just the CPU. (That may or may not help,
depending on the detailed cause of the overheating.)

You could also put time.sleep (5) calls in your code – perhaps in
the loop where you loop over training batches – to give your machine
a chance to cool off a little. (Adjust the 5 by experimentation to find
how much you need to sleep to not overheat.) Of course, this will
slow your training down.

Best.

K. Frank

1 Like
6

Depending on the workload, you might scale down the number of used CPUs via:

MKL_NUM_THREADS=1 OMP_NUM_THREADS=1 python script.py args

and inside your script:

torch.set_num_threads(1)

You might increase the number of threads if this seems to stabilize your system.

1 Like
7

Does num_workers in the Dataloader have any effect on this? I usually use the default num_workers=0. Will changing that help with memory/heating/speed etc?

8

num_workers defines the number of processes used to load each batch in the background, so lowering it might also decrease the pressure on the CPU (however you can’t get lower than 0 workers, which will use the main process).

1 Like
9

On a Side note:
Clean the internals of your laptop with compressed air. Dust can block air vents and generate more heat. If the laptop is some what old, check if there is enough thermal paste between CPU and heat sink. I had faced similar problem with my desktop PC, i had fixed by reapplying thermal paste.

1 Like
10

Concur with the idea to clean with compressed air (even without opening the laptop), pointing at the dissipators. If you are doing ML with a laptop, cleaning dust should be done at least every month, could be much more frequently depending from how intensely you compute, and how dusty the environment is where the laptop is located.

1 Like