Data Tensor Optimisation

1

Hi,

I used to optimize a data tensor after marking it with ( requires_grad= True) and adding it to the optimizer which is defined in the training loop in the same (file.py).

Now I would like to add to this optimization process another data tensor in another different (file.py), how can I add it to the same optimizer?

Thanks for your help

2

If you have defined the other tensor in e.g. a function from the other file, you could import this function and get the tensor in your current script.

Let me know, if I misunderstood your use case.

3

Hi,

Thanks a lot for your information, I have I have tried to make it and included that tensor in the optimization process. In the part of the attached code, that data tensor that I wanted to include in the optimization process is the (data_words) similarly to (data_phrases). Data_phrases is optimized successfully and data is updated, so that when I receive the output tensor ( New_phrases_optimized) data is clearly different. However, for (data_words) it is never optimized, so the input data is exactly the same as the output data. Is there any mistake here?

(reminder: the other part of the code is about softmax regression for the phrases, so that the purpose of having words here is only to include in the optimisation process). Should I add (data_words) to the optimizer in a different way??

for epoch in range(num_epochs):

        print("epoch no. ", epoch + 1)    # training loop
        train_loss = 0
        self.model.train()
        New_words_optimized=[]
        New_phrases_optimized=[]
        for batch_ndx, sample in enumerate(zip(TrainLoader,WordsLoader)):
            self.optimizer.zero_grad()
            data_phrases = Variable(sample[0][0], requires_grad=True)
            data_words = Variable(sample[1][0], requires_grad=True)
            self.data_phrases  = nn.Parameter(data_phrases)
            self.data_words = nn.Parameter(data_words)
            self.optimizer.add_param_group({"params": data_phrases})
            self.optimizer.add_param_group({"params": data_words})
            labels_phrases = sample[0][1].type(torch.LongTensor)-1
            logits, probas = self.model(data_phrases .float())
            loss = self.criterion(logits, labels_phrases) + self.regularizer
            loss.backward(retain_graph=True)
            self.optimizer.step()

            if epoch + 1 == num_epochs:

               phrases_optimized = list(self.parameters())[0].clone()
               words_optimized = list(self.parameters())[1].clone()
               New_phrases_optimized.append(phrases_optimized.detach().numpy())
               New_words_optimized.append(words_optimized.detach().numpy())
            
            train_loss += loss.item()

Thanks a lot

AA

4

Could you check the .grad attribute of data_words after the backward call?

Also, Variables are deprecated since PyTorch 0.4.0, so you can use tensors directly now.
The usual approach would be to pass the parameters as a list to the initialization of your optimizer, but your approach seems to work for data_phrases.

5

Hi

I checked and the tensor of data_words gives “None”, so it was clear that it is not added to the optimizer appropriately. So I came to these 2 lines:

self.optimizer.add_param_group({“params”: data_phrases})
self.optimizer.add_param_group({“params”: data_words})

and replaced (data_phrases) with (data_words) and when I checked the gradient of (data_words) it was showing values and not “None” while (data_phrases) was the one whose .grad gives “None” this time.

So I guess I need to pass the parameters to the optimizer in a different way? May be in one instruction together?

Thanks in advance for your help

Amir

6

Try to pass them as a list during initialization:

optimizer = optim.SGD([data_phrases, data_words], lr=1.)
7

Thank you, the problem is that the intialization step is

self.optimizer = O.optimizer(self.model.parameters(), lr, momentum, optimizer_name)

that i have before the training loop. If I use data_phrases and data_words before they are being defined, there might be an error.

Cant I do something like “self.optimizer.add_param_group({“params”: [data_phrases, data_words]})” in the training phase? I tried that but still not working!

Any advise?

Thanks a lot