How to do transforms, and DataLoading from a CSV file (MNIST Digit Recognition)

#1

Hi,
I am trying to learn PyTorch. So, I started with MNIST Digit Recognition. I am using the CSV dataset file.

I wanted to write my program using functions. Like, I call to def load_test_data and the output of this function must be the processed train images and labels.

After, loading the CSV file into a panda framework I converted it into NumPy and tensor respectively. But, as a newcomer, when I am trying to use transforms it is giving different types of errors. I have searched Google, StackOverflow, and PyTorch Forum and found that I have to create own class to transform. It seems difficult to me!

Is there any way to do transforms and DataLoader for my code?

def load_train_data():
    # Loding CSV file in a dataframe with values
    train_img_df = pd.read_csv(os.path.join(config.dataset_path(), "mnist_train.csv"))
    
    #Getting image pixels values and labels from dataset 
    train_images = train_img_df.iloc[:, 1:]
    train_images = train_images.to_numpy()
    train_images = train_images.reshape((-1, 1, 28, 28))
    train_images = torch.tensor(train_images)
    print(train_images.shape) #torch.Size([42000, 1, 28, 28])
    
    train_labels = train_img_df.iloc[:, 0]
    train_labels = train_labels.to_numpy()
    train_labels = train_labels.reshape(train_labels.shape[0], 1)
    train_labels = torch.tensor(train_labels)
    print(train_labels.shape) #torch.Size([42000, 1])
    
    
    transformations=torchvision.transforms.Compose([torchvision.transforms.ToTensor()])
    #What I have to write in here to transform and for DataLoader
    train_images = ???
    
    #Trying to creat  Dataloder
    train_img_loader = torch.utils.data.DataLoader(train_images, batch_size= config.batch_size)
    train_lab_loader = torch.utils.data.DataLoader(train_labels, batch_size= config.batch_size)
#2

Image transformations in torchvision.transforms work mostly on PIL.Images, so you would need to transform the tensors in train_images to Images first, apply the transformations, and transform the Images back to tensors.
E.g. this might work:

trans = transforms.Compose([
    transforms.ToPILImage(),
    transforms.RandomCrop(24),
    transforms.ToTensor()])
#3

@akib62 I am also trying to learn pytorch on MNIST. Could you please show, where u got MNIST in csv format. Thanks

#4

@angiro you can download the dataset from Kaggle!

#5

@angiro Yes

Link https://www.kaggle.com/c/digit-recognizer/data

1 Like
#6

Hello, @ptrblck Sorry, I am asking you another question! When I am trying to print some images from the test loader but it is giving me an error.

TypeError: Invalid shape (1, 28, 8) for image data

However, my train loader is working fine. Cold you tell me, where I am doing wrong?

test_images = (test_img_df.iloc[:,:].values).astype('float32')
test_images = test_images.reshape(test_images.shape[0], 28, 28)

#Converting Images to tensor
test_tensor = torch.tensor(test_images)/255.0

#Train DataLoder Generator
test_loader = DataLoader(test_tensor, batch_size= config.batch_size, 
                              num_workers = 2,shuffle= True)

#Plot some sample images using the data generator
for data in test_tensor:
    img_grid = make_grid(data[0:8,].unsqueeze(1), nrow=8)
    break

plt.imshow(img_grid.numpy().transpose((1,2,0)))
plt.rcParams['figure.figsize'] = (10, 2)
plt.show()
#7

The error is most likely raised by matplotlib in plt.imshow, which expects an image array in the shape [height, width, channels] or [height, width], while you seem to pass a single channel image array containing the batch dimension to this call.
Remove the batch dimension e.g. via tensor.squeeze(0).