I have a sequence classification problem that I am trying to solve using some kind of “curriculum learning”, which I describe in the following. My dataset consists of sequences and (in the particular problem I am trying to solve) the larger the sequences are, the harder they are to classify correctly.
Therefore, I am scheduling the sequence length in the dataset so that sequences are initially small and their length increases progressively during training. Each mini-batch obtained from the DataLoader has the shape (N, L_i, D), where N is the batch size, L_i is the sequence length at the i-th epoch and D is the dimension of the data.
In order to use the GPU memory efficiently, I would like to use a variable batch size, which is initially larger and decreases along the training epochs, to compensate for the extra memory required by the larger sequences. I thought it would be enough to do the following immediately before each epoch starts:
train_loader.batch_size = N_i,
where N_i would be the desired batch size for the i-th epoch. However, apparently, the DataLoader does not allow the batch size to be changed after initialization, since the following error appears:
ValueError: batch_size attribute should not be set after DataLoader is initialized
Is there any workaround for this limitation?
