I’ve met something similar. Could you take a look at this Possible data parallel memory leak for siamese network ?