Strange very slow training on the server

Its mainly caused by IO problem. It seems that you used a super computer, but the swap area is too small. When loading data from disk, it would comsumes too much memory for data buffering.

Here is some suggestions:

  1. mapping the input feature to output label one by one and contiguously write to disk.
  2. clean the swap area.
  3. use other machine if possible.