Custom data loader and label encoding with CIFAR-10

1

Hi,
I am trying to use a Dataset loader in order to load the CIFAR-1O data set from a local drive.
For learning purposes, I do NOT wish to use the already available loader as shown here:

E.g. torchvision.datasets.CIFAR10.

I downloaded the data manually from here: CIFAR-10 - Object Recognition in Images | Kaggle

Few questions:

  1. Using the original example, I can see that the original labels, are NOT one hot encoded, do I assume correctly that
    cross-entropy and neg. log-likelihood losses in pytorch do NOT require one-hot encodings?

  2. In my custom code (https://github.com/QuantScientist/Deep-Learning-Boot-Camp/blob/master/day%2002%20PyTORCH%20and%20PyCUDA/PyTorch/19%20PyTorch%20CIFAR-10.ipynb), I can see that the data as well as the labels are loaded correctly, when I tried (wrongly?) to one hot encode them, an exception was thrown during iteration.

However, now I do not one hot encode them (assuming 1 above is true), and the following exception is thrown while iterating the dataset:

TypeError: ufunc 'add' did not contain a loop with signature matching types dtype('S55') dtype('S55') dtype('S55')

This is the code for iterating:

imagesToShow=4
for i, data in enumerate(train_loader, 0):
lgr.info('i=%d: '%(i))
images, labels = data
num = len(images)

ax = plt.subplot(1, imagesToShow, i + 1)
plt.tight_layout()
ax.set_title('Sample #{}'.format(i))
ax.axis('off')

for n in range(num):
    image=images[n]
    label=labels[n]
    plt.imshow (GenericImageDataset.flaotTensorToImage(image))
    
if i==imagesToShow-1:
    break

Thanks for any help,

2

yes,they don’t require

I have a rough look at your code, the bug seems come from self.X_train, self.X_train[index] is not string

1 Like
3

Thanks,
Had several other issues,
I have a basic working version here:
https://github.com/QuantScientist/Deep-Learning-Boot-Camp/blob/master/day%2002%20PyTORCH%20and%20PyCUDA/PyTorch/19%20PyTorch%20CIFAR-10.ipynb

image
image710×446 28.4 KB

Now if my batch size is larger than 1, the images wont display :slight_smile:

4

torchvision.utils.make_grid would help