I want to optimize my init_tensor instead of optimizing the model. My data loader train_dataloader gives the initial value of init_tensor. model.eval() for inputs in train_dataloader: init_tensor = inputs["init_tensor"] optimizer = optim.SGD(init_tensor, learning_rate) output = model…

What’s wrong with initilizing the optimizer inside training loop? In the end if you want to optimize the input you will need to initilize an optimizer per input I would say. Once you have optimized the input, old optimizer is no longer useful. If you want to optimize both, model and init tensor, …

But when we initialize the optimizer as per above mentioned, optimizer could only take one step per input_tensor. Thus would you recommend taking multiple step for each input_tensor until a condition as shown below: model.eval() for inputs in train_dataloader: init_tensor = inputs["init_tensor…

I mean, as far as I understand, if you are optimizing a tensor that means somehow that tensor is gonna “remain” during several iterations. It’s a bit diff to discuss about this not knowing what are you trying to do but yes, iterating over that tensor until you reach your target would be the expecte…

To me, what you are doing is a gradient flow. I would suggest you to have a look at the very good post on Wasserstein gradient flow here : https://www.kernel-operations.io/geomloss/_auto_examples/comparisons/plot_gradient_flows_2D.html#sphx-glr-auto-examples-comparisons-plot-gradient-flows-2d-py In…

Thank you @JuanFMontesinos and @tux . In my above code snippet, I have done model.zero_grad(), but I don’t think that is correct. As I’m optimizing init_tensor, I should zero grad the optimizer as below: model.eval() for inputs in train_dataloader: init_tensor = inputs["init_tensor"] # tensor …

In fact, you should zero grad both, model and tensor. Realize that you are doing: while(loss > threshold): output = model(input) loss = loss_fn(output) optimizer.zero_grad() loss.backward() optimizer.step() That way gradients are being accumul…

Thanks @JuanFMontesinos . Cleaning it up, the final pipeline will look like this: model.eval() for batch_id, inputs in enumerate(train_dataloader): init_tensor = inputs["init_tensor"] # tensor to be optimized optimizer = optim.SGD(init_tensor, learning_rate) input_tensor = inputs["inp…

According to the link below, we don’t need to do model.zero_grad() as the model is in eval mode. [image] Backprop in PyTorch? Hi, So pytorch is very different from tensorflow in the sense that your don’t create the graph once, and then just backprop through it. The who…

Did you find a solution/explanation for this?

Add one tensor from dataloader to optimiser

alwynmathew (Alwyn Mathew) February 6, 2020, 9:30am 9

According to the link below, we don’t need to do model.zero_grad() as the model is in eval mode.