Oh ok. I understand now.
version 0.4.1: Maybe, pytorch uses cuda:0 always by default?
pytorch master 0.5.0a0+ab6afc2: I have verified that this issue is fixed.
Sorry for many edits of the answer 
1 Like