I write a small net and train it on a very simple problem, but sometimes I find that the speed of pytorch becomes very slow.
I say SOMETIMES, because I can’t find the pattern of this behavior. I changed some parameters, for example, leaning rate or the number of layers of the net, this issue disappeared. And I changed those parameters back to their old settings, the slow-down problem didn’t appear again. So I don’t know what’ the problem.
If I press Ctrl+C when the program get stuck, I find it usually stops at this line
return F.linear(input, self.weight, self.bias)
Below is what I get If I press Ctrl+C
Traceback (most recent call last):
File "/mnt/e/workdir/ML_ADS/gauss/diff_min/diff_sin.py", line 134, in <module>
x0_2 = model(x, x+noise, q)
^^^^^^^^^^^^^^^^^^^^
File "/mnt/d/software_install/pytorch/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/d/software_install/pytorch/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/e/workdir/ML_ADS/gauss/diff_min/diff_sin.py", line 115, in forward
x0_2 = self.dirformer(z, x)
^^^^^^^^^^^^^^^^^^^^
File "/mnt/d/software_install/pytorch/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/d/software_install/pytorch/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/e/workdir/ML_ADS/gauss/diff_min/diff_sin.py", line 106, in forward
return self.mlp(zx)
^^^^^^^^^^^^
File "/mnt/d/software_install/pytorch/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/d/software_install/pytorch/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/d/software_install/pytorch/lib/python3.12/site-packages/torch/nn/modules/container.py", line 217, in forward
input = module(input)
^^^^^^^^^^^^^
File "/mnt/d/software_install/pytorch/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/d/software_install/pytorch/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/d/software_install/pytorch/lib/python3.12/site-packages/torch/nn/modules/linear.py", line 116, in forward
return F.linear(input, self.weight, self.bias)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt