Hello! I’m working on GCN recently, I build a dataset myself, consisting of about 150,000 datas with a data type of torch_geometric Data. Specifically, I rewrite Data class simply and add another attribute z which is a extra target as my model have two targets:
from torch_geometric.data import Data
class MyData(Data):
def __init__(self,x,y,z,edge_index,pos):
super(MyData, self).__init__(x=x,y=y,edge_index=edge_index,pos=pos)
self.z = z
I save my datas(named as sample) one by one simply using torch.save:
torch.save(sample, 'new_graph_data/{}.pt'.format(str(image_id)+'_'+str(gt_id)))
Therefore, I have got 150,000 pt-files, leading to EXTREMELY slow speed when I try to loading my datas in a traditional way:
all_data_path = os.listdir('F:/CODE/Pytorch/Diplomer/new_graph_data/')
train_ = 5000 # for testing
test_ = 6000
train_data_path = all_data_path[:train_]
test_data_path = all_data_path[train_:test_]
class MyOwnDataset(Dataset):
def __init__(self, root, mode):
super(MyOwnDataset, self).__init__()
self.root = root
self.mode = mode
def get(self, idx):
if self.mode == 'train':
data = torch.load('F:/CODE/Pytorch/Diplomer/new_graph_data/'+train_data_path[idx])
elif self.mode == 'test':
data = torch.load('F:/CODE/Pytorch/Diplomer/new_graph_data/'+test_data_path[idx])
return data
def len(self):
if self.mode == 'train':
return train_
elif self.mode == 'test':
return test_-train_
After researching, I found that the troublesome slow speed results from the loading of huge amount of small files(for I save my pt-file one by one).So I decide to use LMDB to load my files.
txn = env.begin(write=True)
tqdm_iter = tqdm(enumerate(zip(all_file_list, keys)), total=len(all_file_list), leave=False)
for idx, (path, key) in tqdm_iter:
tqdm_iter.set_description('Write {}'.format(key))
key_byte = key.encode('ascii')
data = torch.load(path)
txn.put(key_byte, data)
But it’s known that LMDB requires a data type of bytes, I have no idea how to convert them.
Can u give me some advice?
Or maybe u have other efficient ways to load geometric Data? Any ideas can help me load faster?
Thank u so much!! 