Looping over 2 DataLoaders simultaneously (one for images another for numbers)

Hi everyone,

I am training a CNN to detect a specific type of skin cancer. I also have a series of attributes per image, such as the age and sex of the patient that corresponds to the particular image.

I explain the data flow in the diagram above:

  1. The data from the first Data Loader (images ) flows through the CNN, then gets flattened and passed through the linear layer.

  2. The data from the second Data Loader (attributes) needs to get appended to the flattened structure that goes from a shape of (batch_size, 240) to a shape of (batch_size, 259).

The question is how to loop over the 2 different Data Loaders to achieve this?

#defining trainloader with images
mini_b = torch.utils.data.DataLoader(torch.utils.data.TensorDataset(tX_train, ty_train), 
                                       batch_size=100, shuffle=False) 

#defining trainloader with attributes
mini_b2 = torch.utils.data.DataLoader(torch.utils.data.TensorDataset(ext_data_tensor), 
                                       batch_size=100, shuffle=False) 

#training loop
for i in range(500): 
    #iterating through data, one batch at a time
    for x_j, y_j in mini_b:
        #performing one training step
        out = t_step(x_j, y_j, cnn1, opt1) 
    if i%10 == 0:
        print("Epoch: {}, Loss: {:.4f}".format(i, out))

Have you tried zipping the iterators together?

for b, b_2 in zip(mini_b, mini_b2):
    b_x, b_y = b
    b2_x, b2_y = b_2

Thanks a lot for your swift reply! Here, the for loop still operates one batch at a time, correct ?

Yes, as soon as you make sure that your items are aligned properly, zip will do 1 iteration for the first one and one for the second and then return the result as a tuple.

It worked! Thanks a lot @zarkopafilis :love_you_gesture:

1 Like