I am working with some neural network models which I want to combine. My idea is to train the network with multiple datasets on multiple different losses simultaneously. For example, a batch consists of 8 samples of dataset 1, 8 samples of dataset 2, and 16 samples of dataset 3. I wanted to concatenate the dataset and just write a custom dataloader to ensure this kind of sampling. By doing this, the structure of all the code need not be changed which is a plus. The problem now is that the datasets are of different formats. For example, X1 = (21464, 500, 129), y1 = (21464, 3) and X2 = (30842, 500, 129), y2 = (30842, 2). In this example the labels are the problem. Are there different ways to approach this problem or does concatenating still somehow work?
Thanks for your help!