Pytorch Lsun dataset

vision
1

My code does not continue at
with self.env.begin(write=False) as txn:
self.keys = [key for key, _ in txn.cursor()]

my output is :
length: 168103
1
my code is :
class LSUNClass(data.Dataset):
def init(self, db_path, base_size=64,
transform=None, target_transform=None):
import lmdb
self.db_path = db_path
self.env = lmdb.open(db_path, max_readers=1, readonly=True, lock=False,
readahead=False, meminit=False)
with self.env.begin(write=False) as txn:
self.length = txn.stat()[‘entries’]
print('length: ', self.length)
cache_file = db_path + ‘/cache’
if os.path.isfile(cache_file):
self.keys = pickle.load(open(cache_file, “rb”))
print(‘Load:’, cache_file, 'keys: ', len(self.keys))
else:
print(“1”)
with self.env.begin(write=False) as txn:
self.keys = [key for key, _ in txn.cursor()]
print(“3”)
pickle.dump(self.keys, open(cache_file, “wb”))
Actually my code is like stopping at
with self.env.begin(write=False) as txn:
self.keys = [key for key, _ in txn.cursor()]
I did not solve this problem.
Have you any idea?

2

What is txn doing and is the code working without using this library?
If so, try to use num_workers=0 in your DataLoader (if not already done) and rerun it.
If that’s working, the txn lib might run into some multiprocessing issues with the PyTorch DataLoader.

PS: You can add code snippets by wrapping them into three backticks ```, which makes debugging a bit easier. :wink:

3

First of all, thank you for your response.
I do not know what txn’s function is in the code since I found this code on internet. However, when I make a changing that you say, code is not working.

5

Again thank you for your interest and advices.
I have just solved the problem. The problem is not related to wrong implementation, it is about google colab. I searched the problem online deeply and I saw that for huge dataset this problem may occur in google colab when reading dataset from google drive. I upload my dataset into google colab itself, not into drive. Then problem does not exist. Thank you :slight_smile: