Merging all tensor files into one large tensor file

1

Hi,

I have 10 PyTorch tensors saved as 1.pt, 2.pt, …, 10.pt (each file is ~ 3GB). I want to create a training dataset from all of them.

  1. One way to do this (inefficient way though) is to create a list of tensors and then use torch.stack on the list to create a large tensor. I run into the CUDA memory issue because of the large file size.

  2. Other way is to load each file into separate batches but I am not sure how to actually implement this.

Thanks!