Manual Streaming?


I’m having a hard time with a tight deadline trying to get a custom data set to load correctly. I have thus created a label, numerical column, categorical column and image tensors separably. However, when I load them into the GPU, I get a memory error.

Is there a way a manually stream batches out of the tensors into the gpu, 10 at a time?

What kind of “memory error” are you getting?
Could you post the error message with the stack trace, please?

Hi @ptrblck. I already ran a process where I pulled all of the data prep out of the custom data set to include transformation to tensors.

Now my custom data set looks like this:

class tensor_data(Dataset):
    def __init__(self, num, cats, labels, images):
        self.num = num_tensor
        self.cats = cats
        self.labels = labels
        self.images = images
    def __len__(self):
        return len(self.num)
    def __getitem__(self, idx):
        return self.num[idx], self.cats[idx], self.labels[idx], self.images[idx]

That said, it’s also solved the single image per batch issue and it seems to be running faster. It’s just as easy to define a function to do the prework so I’m happy. Thank you, as always, for the great support. The community is lucky to have you.