Empy parameters in optimizer.param_groups

1

I have encountered a strange behavior. The following piece of code works as expected

model = models.resnet152(pretrained=True)

params_to_update = [{'params': model.fc.parameters(), 'lr': 0.001}]

optimizer = optim.Adam(params_to_update) 

print(optimizer.param_groups)

However if I do something like this:

params_to_update = [{'params': model.fc.parameters(), 'lr': 0.001}]

for d in params_to_update:
    for param in d['params']:
        print(param.requires_grad)

optimizer = optim.Adam(params_to_update) 

print(optimizer.param_groups)

I get:

True
True
[{'params': [], 'lr': 0.001, 'betas': (0.9, 0.999), 'eps': 1e-08, 'weight_decay': 0, 'amsgrad': False}]

where the 'params': [] is empty now (before inserting the for-loops it was not empty). I find it very strange since I haven’t altered the params_to_update dictionary and just made some print statements in the for-loop.

2

Try

params_to_update = [{'params': list(model.fc.parameters()), 'lr': 0.001}]

Best regards

Thomas

3

Thank you, that works! However I was wondering why is that happening and why using the list helps?

4

Without the list cast you have an iterator that you can only consume once. If you do that in the for loop, there won’t be anything left when you pass it to the optimizer.

Best regards

Thomas

1 Like
5

Hi, is this a general property of all iterators or is it something that’s special for model.parameters() ?

6

No that’s iterators in general. I believe that if one would want to avoid that, you’d need something like generators (e.g. what datasets do), where you can get multiple (“rewound to start”) iterators from a generator, but then you’d probably have other drawbacks.

1 Like