Loss.backward() call not updating loss

autograd
1

Hi there, I’m training text features recently, but the loss doesn’t updated, and I really don’t know where the problem is

here is the code

        for iter in tqdm(range(100)):
            total_style_loss = 0
            for j in range(i):
                style_prompts_i = model("style_prompts", i)
                style_prompts_j = model("style_prompts", j)
                sim = cos_sim(style_prompts_i, style_prompts_j)
                total_style_loss = total_style_loss + sim
            loss_style = total_style_loss / i

            optimizer.zero_grad()
            loss_style.backward()
            optimizer.step()

the result is:

tensor([0.9084], device='cuda:0', grad_fn=<DivBackward0>)
  1%|▊                                                                                   | 1/100 [00:00<00:23,  4.30it/s]tensor([0.9084], device='cuda:0', grad_fn=<DivBackward0>)
tensor([0.9084], device='cuda:0', grad_fn=<DivBackward0>)
tensor([0.9084], device='cuda:0', grad_fn=<DivBackward0>)
tensor([0.9084], device='cuda:0', grad_fn=<DivBackward0>)
  5%|████▏                                                                               | 5/100 [00:00<00:05, 17.29it/s]tensor([0.9084], device='cuda:0', grad_fn=<DivBackward0>)
tensor([0.9084], device='cuda:0', grad_fn=<DivBackward0>)
tensor([0.9084], device='cuda:0', grad_fn=<DivBackward0>)
tensor([0.9084], device='cuda:0', grad_fn=<DivBackward0>)
  9%|███████▌                                                                            | 9/100 [00:00<00:03, 24.49it/s]tensor([0.9084], device='cuda:0', grad_fn=<DivBackward0>)
tensor([0.9084], device='cuda:0', grad_fn=<DivBackward0>)
tensor([0.9084], device='cuda:0', grad_fn=<DivBackward0>)
tensor([0.9084], device='cuda:0', grad_fn=<DivBackward0>)
 13%|██████████▊                                                                        | 13/100 [00:00<00:03, 28.87it/s
2

Can you check:

  • are the .grads being populated before .step()
  • is style_prompts_i.requires_grad True
  • are your parameters requires_grad=True
3

Thank you for your answer :smiley: