How do I successfully unpack values from batches?

vision
1

Hello:

I am running images through an encoder in batches of 10 in order to get vectors back. The data loader unpacks the ID, image and label (even though I’m not using the label). After running the image through the encoder, I append the vector to a list and the ID to another list. I would like to append each image and ID to the list by themselves but the code is appending lists of 10 ids and 10 vectors. How can i append the vectors and IDs separately and not batches of 10?

Below is my code:

epochs = 1
max_train_batch = 6361
max_test_batch = 1559

policy_list =
test_policy_list =
train_output =
test_output =

for i in range(epochs):
model.eval()
for b, (image, label, policy) in enumerate(train_loader):
policy_list.append(policy.data.cpu().numpy())
image = Variable(image).cuda()

    with torch.no_grad():
        output = model(image)
        train_output.append(output.data.cpu().numpy())

    if b == max_train_batch:
        break
        
    b += 1

    print(f'train_set, epoch: {i}, batch: {b}')
    
    #run test set thru
for b, (image, label, policy) in enumerate(test_loader):

    test_policy_list.append(policy.data.cpu().numpy())
    image = Variable(image).cuda()

    with torch.no_grad():
        output = model(image)
        test_output.append(output.data.cpu().numpy())

        if b == max_test_batch:
            break

        b += 1

        print(f'test_set, epoch: {i}, batch: {b}')
2

torch.split will get you a list of tensors.

Best regards

Thomas

3

Thanks Thomas. I tried the following and got an error.

for b, (image, label, policy) in enumerate(train_loader):
policy_list.append(policy.split.data.cpu().numpy())
image = Variable(image).cuda()

    with torch.no_grad():
        output = model(image)
        train_output.append(output.split.data.cpu().numpy())

The policy (ID) and image are tensors so should I split them, save those to a variable then append them to my list?

4

After reading the docs, it seems this method results in a tensor. I want the ID and/or outcome of the model to be a float value that is appended to my list, so I can later turn those into a dataframe to be joined to a tabular data set.