Basic quick question, size of dataset object

vision
1

Hi

sorry this is a really basic question but I genuinely can’t figure it out. I have a custom dataset object and had to limit my batch size so would all fit on the GPU ram. But what I can’t figure out is what size the dataset object actually is? I tried this:

sys.getsizeof(myDataset)

and it returned-

56

As I understand that 56 bytes? This doesn’t seem right as its a ton of images. Can anyone enlighten me?

Ian

2

Depending how you are loading the data (eager or lazily) you might be able to check the number of elements the entire data holds and calculate the size with it.
Currently you might be checking the size of the Python object itself.

If I understand your use case correctly, you are interested to see how much memory a batch would use?
If so, this should work:

# get the batch from the DataLoader
batch = torch.randn(16, 3, 224, 224)
print('{}MB'.format(batch.nelement() * batch.element_size() / 1024**2))
> 9.1875MB
1 Like