I’m also having the same CPU memory issue with LSTMs, strangely it is only affected if the batch size and hidden size are above a certain level. In the example code below, the memory measurement is on the Python process directly.
https://discuss.pytorch.org/t/lstm-cpu-memory-leak-on-specific-batchsize-and-hidden-size/89135