Efficient dataset saving/loading for large PyG graphs (8000 patients or more)

data
1

Hi everyone

I am building a large torch_geometric dataset with ~8,000 object(.pt) , are patients in my case.
Each patient contains several tensors, producing a Data object (node features, edge indices, and a single target value).

Originally, I saved each graph as a separate .pt file, so I had ~2000 files for a smaller test case(around 200 GB). This was easy to load but took too much storage space.

I tried to reduce it, saving batches of 20 graphs into a single compressed file (.pt.xz).

the reduction of space was drastic ! but loading and training was impossible.

What’s the recommended PyTorch/PyG-friendly way to:

Store large datasets of graphs efficiently (compression + batching)?

Load them quickly in getitem without hanging?

Keep global indices consistent after batching?

Thank you

any advice is welcome

2

After many attempts, I finally found an effective way to compress my dataset without slowing down the training phase. I now save batches of 20 graphs into a single compressed file using LZ4 compression (.pkl) with joblib.dump, and later load them with joblib.load. This approach drastically reduced the dataset size—from 1TB down to 80GB."