Making minibatches with collection.deque() fails to reach all the examples given a batch_size


l would like to make minibatches. However, l noticed that collection deque reaches all the examples only if
rest of division is equal to 0 between number of examples and batch size.

For instance if l have 50 examples and l set my batch_size to 4, then l get 12 mini batches of 4 examples each . 12*4=48 BUT it remains 2 examples not processed 49 and 50

How can l solve that ?

Here is my code

import collections
indices_test = collections.deque()

while len(indices_test) >= batch_size:
        batch_idx_test = [indices_test.popleft() for i in range(batch_size)]
        test_x, test_y=test_data[batch_idx_test, :], test_labels[batch_idx_test]
        test_x = Variable(torch.FloatTensor(test_x).type(dtypeFloat), requires_grad=False)
       output =forward(test_x)

Thank you

That’s normal if the number of samples isn’t a multiple of the batch size. What you can do is either use partial mini-batches or shuffle the dataset at each epoch, so that samples you ignored at one epoch will likely be used in the following ones. Also, have a look at DataLoader, that simplifies some of these things for you.