I wonder the training scenario when I train a model jointly over multiple datasets

1

I have multiple datasets, say 2 datasets.
And they have different training scheme and different labels.
So, I think I can’t concatenate 2 datasets into single dataset.

So, I’m thinking to create 2 custom dataset classes.

Then, what is the best (or correct) way to train a single model?

# Let's say epoch is 10
for i in range(10):
  # For example, I first train network over one whole dataset
  # (like using 10000 pairs of images)
  for imgs,lbls enumberate(one_data_loader):
    train_network(imgs,lbls)
  # Then, I should train network over another whole dataset?
  # (like also using 10000 pairs of images)
  for imgs,lbls enumberate(second_data_loader):
    train_network(imgs,lbls)

Is this correct training scenario?
Because I guess if I train network over one large dataset,
it means parameters are tuned based on that dataset,
and if I train tuned network over different large dataset,
I guess current network is tuned based on current dataset,
with disappearing effect of training over first dataset.

I wonder whether my guess is correct.
If so, I wonder what’s the correct training way for this case.

2

Well if you want to do shuffling, just generate half of the images from the first dataset and other half from the other dataset and pass then to your model.

If your batchsize=32. Get 16 images from first dataloader and 16 images from second dataloader and pass them to your network, independently or shuffled.