Setting num_workers=0 actually showed the real problem which simply was :
AttributeError: 'SVHN_Dataset' object has no attribute 'is_train_split'
I simply had a misspelling and that was causing the issue. basically as noted here and here :
this error only occurs when you try to do multiprocessing on some code with errors in it.
hope this is useful for somebody out there!