Do you mean that if you execute X times the first pipeline you are obtaining the same result? I suppose that you did not change the order of init of your model, dataloaders, right? Without more information about your network and your code it is difficult to me to know what is the problem. If you provide a piece of code to reproduce this behavior I can take a look.
Moreover, I think it will be a good idea to set cudnn.benchmark to false if your input sizes changes at each iteration. cudnn.benchmark choose the best algorithm for your input data. That can be a source of randomness. It is quite possible the cudnn lstm kernels are not deterministic as it is said here.