I have also been working with padded sequences and the need to order them. I am currently running into a problem where I have 2 different inputs and 2 RNNs. I can sort both inputs according to their length, but then the respective dimensions don’t match each other anymore, so I have to re-instantiate the original order after running through the RNN (basically as shown here: RNNs Sorting operations autograd safe?). I’m just worried that this operation is not autograd-safe, i.e. that the gradients get lost or are associated with the wrong matrix entries after resorting.
Do you know about similar problems or can help?