Merging all tensor files into one large tensor file


I have 10 PyTorch tensors saved as,, …, (each file is ~ 3GB). I want to create a training dataset from all of them.

  1. One way to do this (inefficient way though) is to create a list of tensors and then use torch.stack on the list to create a large tensor. I run into the CUDA memory issue because of the large file size.

  2. Other way is to load each file into separate batches but I am not sure how to actually implement this.