Hi there,
I tried two methods to generate the same features using a custom dataloader:
-
Prepare the features beforehand and save it to disk as pickle file, and load the pickle file directly in the “_getitem_” function.
-
Generate the features on the fly in the “_getitem_” function, this process involves large matrix operation using NumPy.
Both of the final returned data are the same shape and type, the only difference is the first one is loaded from disk while the second one is calculated on the fly with some matrix operation. However, the CUDA memory usage of the second one will be about twice the first one, without decreasing the minibatch_size, I will got “RuntimeError: CUDA out of memory”. Does it mean that the dataloader will take up GPU during the feature process?
I tried to fix this issue by setting the pin_memory to False, delete some intermediate variables in the “_getitem_” function, but none of them worked.
Any insight appreciated!