Hello,
I have a weird issue and I can’t seem to find a typo. I have a training data set of shape (3719,100) and a testing data set of (6635,100). When I transform all data to tensors, the custom dataset and loader for the test data seems to be loading train data according to the .size(0 outputs. Below is the code I’m working with:
class tensor_data(Dataset):
def __init__(self, num, cats, labels, images):
self.num = num_tensor
self.cats = cats
self.labels = labels
self.images = images
def __len__(self):
return len(self.num)
def __getitem__(self, idx):
return self.num[idx], self.cats[idx], self.labels[idx], self.images[idx]
Here is how I derive the data to use the above:
###########################train tensors#####################################
num_tensor = torch.tensor(np.asarray(train_sample.loc[:, numerical_columns]), dtype = torch.float)
col_tensor = torch.tensor(np.asarray(train_sample.loc[:, non_loca_cat_columns]), dtype = torch.long)
label_tensor = torch.tensor(np.asarray(train_sample.loc[:, 'target']), dtype = torch.long)
image_tensor = []
for i, row in train_sample.iterrows():
image_tensor.append(train_transform(Image.open(Path(row['location']))))
images = torch.stack(image_tensor)
print("train: ", num_tensor.size(), col_tensor.size(), label_tensor.size(), images.size())
output: train: torch.Size([3719, 6]) torch.Size([3719, 8]) torch.Size([3719]) torch.Size([3719, 3, 224, 224])
train_data = tensor_data(num_tensor, col_tensor, label_tensor, images)
print('train_data:', len(train_data))
output: train_data: 3719
train_loader = DataLoader(train_data, batch_size = 10, shuffle = True)
print('train_loader: ', len(train_loader))
output: train_loader: 372
So all of the above makes sense. This is what happens when I do the same with the test data.
test_num_tensor = torch.tensor(np.asarray(test.loc[:, numerical_columns]), dtype = torch.float)
test_col_tensor = torch.tensor(np.asarray(test.loc[:, non_loca_cat_columns]), dtype = torch.long)
test_label_tensor = torch.tensor(np.asarray(test.loc[:, 'target']), dtype = torch.long)
test_image_tensor = []
for i, row in test.iterrows():
test_image_tensor.append(test_transform(Image.open(Path(row['location']))))
test_images = torch.stack(test_image_tensor)
print("test: ", test_num_tensor.size(), test_col_tensor.size(), test_label_tensor.size(), test_images.size())
output: test: torch.Size([6635, 6]) torch.Size([6635, 8]) torch.Size([6635]) torch.Size([6635, 3, 224, 224])
test_data = tensor_data(test_num_tensor, test_col_tensor, test_label_tensor, test_images)
print('test_data:', len(test_data))
output: test_data: 3719
test_loader = DataLoader(test_data, batch_size = 10, shuffle = False)
print('test_loader: ', len(test_loader))
output: test_loader: 372
The above shows the test tensors the correct size of (6635, x) but when bringing them in through the custom dataset then loader, I get the lengths of the train loader. Am I missing something simple?