Hard drive has rhythmic noise when training model

Low-frequency noise when CUDA usage at a higher level in the above image, and whenever the usage drops down (2 valleys in the image), the noise change to a higher frequency. These shifts happen periodically when the training starts and then end immediately when training is over.

What might be the problem?

When I set a larger batch size, the noise frequency becomes lower too. When the batch size is large enough, the noise eventually disappeared.

What the heck?

Are you sure your hard drive is making this noise? I’m pretty sure my GPU makes different noises based on the workload and it seems Alec Radford can even distinguish different model architectures by the sound of his GPU :wink:

1 Like

@ptrblck :open_mouth:

Actually I’m not sure for I haven’t open up my PC to check exactly. How come the gpu can make such noise? The fans?

ptrblck via PyTorch Forums noreply@discuss.pytorch.org于2019年3月5日 周二17:13写道:

I have no idea what exactly is responsible for the sounds, but mine is sometimes quite loud so that you can even “hear” each iteration.

EDIT: Maybe @ngimel might have some insight here. :wink:

Yes, I can even make descent guess on the batch size given the frequency of the sound lol

ptrblck via PyTorch Forums noreply@discuss.pytorch.org于2019年3月5日 周二22:37写道:

1 Like