Value Error :Cannot take sample than population when 'replace=false'

Following is the error trace:
File “main.py”, line 267, in main
_ = train(iteration,optimizer)
File “main.py”, line 134, in train
for j, (img,fix,img_seq,target) in enumerate(trainloader):
File “/DATA/rani.1/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 345, in next
data = self._next_data()
File “/DATA/rani.1/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 385, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File “/DATA/rani.1/.local/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py”, line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File “/DATA/rani.1/.local/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py”, line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File “./util/data_processing.py”, line 275, in getitem
for i,idx in enumerate(np.random.choice(np.arange(len(pt_img)),self.pt_len,replace=False)):
File “mtrand.pyx”, line 1168, in mtrand.RandomState.choice
ValueError: Cannot take a larger sample than population when ‘replace=False’

Based on the stack trace it seems you are using a custom Dataset, which seems to be defined in ./util/data_processing.py. Inside the __getitem__ method you are apparently using np.random.choice, which get’s an invalid argument combination.
It seems that len(pt_img) might be smaller than self.pt_len which could yield a similar issue as this code snippet:

np.random.choice(np.arange(10), 10, replace=False) # works
np.random.choice(np.arange(9), 10, replace=False) # fails
2 Likes

Thanks for your patient explanation. But if we encounter this problem, how to solve it?

Thanks!

As seen in the previous example you won’t be able to sample e.g. 10 values out of 9 without replacement.
Either use replace=True or make sure you are not trying to draw more samples than are available.