Hey you could do this :
After making two dataset classes, make a third class called fusion class that contains the two dataset classes.
The __ get __ function calls randomly or alternatively get function of the two constituiting data classes.
See this link for example of how you can do that. You can also create balance between the datasets images to load in that class.
This is probably just bit neater to do! 