Parameters disappear after setting require_grad = True

Hello researchers and engineers, I was working on some stable diffusion related task, and got this problem. The relavent code is as below.

trainable_params = text_encoder.get_input_embeddings().parameters()
print('num of parameter list1:', len(list(trainable_params)))
for param in trainable_params:
    param.requires_grad_(True)
print('num of parameter list2:', len(list(trainable_params)))
optimizer = torch.optim.AdamW(trainable_params, 
                                lr=learning_rate, betas=(adam_beta1, adam_beta2),
                                 weight_decay=adam_weight_decay, eps=adam_epsilon)

At first I directly put text_encoder.get_input_embeddings().parameters() into the optimizer, and got RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn. Then I iterated the trainable_params and set requires_grad_ of every param to True and got the strange behavior. For the two print() functions, the output of the first one is 1 and the output of the second one is 0.

Why is this happening? Would anyone help me solve it or explain the reason? thanks a lot

Isn’t this expected behavior in Python for all iterators? I.e. once you iterate then they are exhausted and you would need to recreate the iterator again? I’m not in front of my workstation to try it out.

Yes, seems to be the expected behavior:

model models.resnet18()
trainable_params = model.parameters()
print('num of parameter list1:', len(list(trainable_params)))
# num of parameter list1: 62
print('num of parameter list1:', len(list(trainable_params)))
# num of parameter list1: 0

a = [0, 1, 2]
a_iter = iter(a)
print(len(list(a_iter)))
# 3
print(len(list(a_iter)))
# 0