Data Augmentation Transforms for Sequences on minority class

1

I have a sequence of images, so with batch size 1, my getitem method output would be of size [1,9,3,256,256] with sequence length 9.
The method currently looks like this:

    def __getitem__(self, idx):

        seq = self.data[idx]
        ims = np.empty((self.seq_length, 3, 256, 256), dtype=np.uint8)
        labels = np.empty((self.seq_length), dtype=int)

        for i, case in enumerate(seq.cases):
            ims[i] = imread(case.path)
            labels[i] = case.cls
                        
        # transform to tensors
        ims = torch.from_numpy(ims)
        labels = torch.from_numpy(labels)

        if self.transform: 
             ims = self.transform(ims)
            
        return ims, labels

Since my data is highly imbalanced, I’d like to perform transformations only on the minority class (labels==1). I know that I have to check for labels[i]==1 but the problem is that in one sequence, the labels can be 0 and 1. I am not 100% sure how torchvision transformations work. Will they change the length of my sequence or can I do it like this:

for i in range(len(ims)):
            if (labels[i] == 1) and self.transform: # check for minority class
                ims[i] = self.transform(ims[i])
2

Your code looks alright and the sequence length should not be changed, since you are reassigning the transformed tensors to ims. Do you see any issues using this approach?

3

Thanks for your reply! Before using data augmentation, my model was overfitting after a few epochs but with data augmentation it is now overfitting after the first epoch already. Do you think this is just caused by the augmentation methods I am using or can it be a result of the way I am applying the augmentation?

4

Your data augmentation might be too aggressive and might not represent the underlying data domain properly, so that the model learns the training set but gets worse in classifying the validation samples.

5

Thanks! Is there anything else I can do to prevent overfitting? I implemented the WeightedRandomSampler as well and now the loss is strictly decreasing for the first 10 epochs but then it always starts to overfit (also without using data augmentation). I am using a pretrained ResNet18 and also tried to freeze all except the last layer.