.cuda() function can't copy list to gpu

#1

Hi,

In my resnet class, I defined a list of layers.

for k in range(4):
self.layers[k] = self.make_layer(…)

When running, it showd “Runtime Error: tensors are on different GPUs”, although I have only one GPU.

I modified the code, added something like:

self.layer1 = self.layers[0]
self.layer2 = self.layers[1]
…

in the init function and ran again, everthing is OK.

I guess current .cuda() function cannot copy the content in complex data structure, such as list, to GPU. Am I right?Preformatted text

#2

you are correct. .cuda() will only copy parameters and buffers of a model on to GPU, not all the other custom attributes.

1 Like
#3

@smth You are really quick! :sweat_smile:

In my case, if .cuda() can copy the parameters of the list of layers, the code can be much nicer, especially when there are many layers. So, is there any solution for my case?

Many thanks!
Ben

#4

Hi,

For list of Modules, you have http://pytorch.org/docs/nn.html#modulelist
And for list of Parameters you have http://pytorch.org/docs/nn.html#parameterlist

Using them will make sure that all methods like .cuda() or .parameters() work as expected !

5 Likes
#5

Thanks @albanD!

Now my code is like this and it works:

self.layer_list = []
for k in range(4):
    self.layer_list.append(self.make_layer(...))
    
self.layers = nn.Modulelist(self.layer_list)

I didn’t use ParameterList. Please tell me if there is any thing wrong！

#6

ParameterList is if you want to store a list of Parameters, not needed in your case.
You can do even better :slight_smile: :

self.layers = nn.Modulelist()
for k in range(4):
    self.layers.append(self.make_layer(...))
3 Likes
#7

Cool! Thanks!
I found some similar posts here.
This should be added to the document and made easy to find. :grin: