Backward() does not work

autograd
1

Dear all,

I have the following code which is used to train a neural function. u(x,y) is my function and I am trying to get it by minimizing the loss (-u’‘(x,y)-f)^2 where u’'(x,y) stands for Laplacian and f as well.
Everything is running fine but I have an issue with the backward() function which does not work here. The loss remains at the exact same level meaning that the neural network does not work.
Can someone help?

Thanks in advance

import torch
import torch.nn as nn
import numpy as np

learning_rate = 0.01
num_epochs = 15

N = 15

U = lambda x: torch.sin(torch.pi * x[0]) * torch.sin(2 * torch.pi * x[1])
f = lambda x, u: (5 * torch.pi**2* torch.sin(torch.pi * x[0]) * torch.sin(2 * torch.pi * x[1]))

x = np.arange(0, 1.1, 1/(N+1))[1:N+1]
p = np.array([])
for i in range(N):
p = np.append(p, x)
q = np.sort(p)
z = np.column_stack((p, q))
input = torch.tensor(z).to(dtype = torch.float32).requires_grad_(True)

class Net(nn.Module):
def init(self):
super(Net, self).init()
self.fc = nn.Sequential(nn.Linear(2, 64), nn.Tanh(), nn.Linear(64, 1, bias = False))

def forward(self, input):
    x = input[0]
    y = input[1]
    input = self.fc(input)
    return input * x * (1-x) * y * (1-y)

model = Net()
gradient = torch.optim.Adam(model.parameters(), lr = learning_rate)

def loss(x, U, f):
nablaU = torch.tensor([0]).to(dtype = torch.float32).requires_grad_(True)
output = model.forward(x)
Uneuronal_d = torch.autograd.grad(output, x, grad_outputs = torch.ones_like(output), allow_unused=True, retain_graph = True, create_graph=True)
nabla_x = torch.tensor([0])
for grad in Uneuronal_d:
grads_inner = torch.autograd.grad(grad, x, grad_outputs = torch.ones_like(x), create_graph=True, retain_graph=True, allow_unused=True)[0]
nabla_x = torch.tensor([torch.sum(nabla_x + grads_inner)])
nablaU = torch.cat((nablaU , torch.tensor([nabla_x])), dim=0)
return(torch.mean(nablaU[1:]))

for i in range(num_epochs):
gradient.zero_grad()
l = loss(input, U, f)
l.backward()
print(l)
gradient.step()

2

Your code is hard to read as it’s not formatted properly.
However, it seems you are detaching tensors implicitly by recreating a tensor in:

nabla_x = torch.tensor([torch.sum(nabla_x + grads_inner)])
nablaU = torch.cat((nablaU , torch.tensor([nabla_x])), dim=0)
return(torch.mean(nablaU[1:]))

nablaU is now a new tensor, without any Autograd history.
Also, nablaU is a newly created leaf tensor also without any Autograd history:

nablaU = torch.tensor([0]).to(dtype = torch.float32).requires_grad_(True)
3

Hi,
Thanks,
Taking into account your comments, it seems to work but I am worried about the loss level.
It has decreased but the level remains high.
It is probably linked to the fact that we are using the second deirvative in the loss. Anyway, I need to recheck it and will revert if need be.
Thank you again for your reactivity

4

No, I don’t think your approach works as intended.
The backward call will not raise an error, since nablaU is differentiable, but since it’s a newly created tensor in the forward method it will never be updated.
Also, since you are detaching nabla_x, the previously used parameters also shouldn’t get any gradients.
Run a single iteration as:

output = model(input)
loss = criterion(output, target)
loss.backward()

and check the .grad attributes of the model’s parameters via:

for name, param in model.named_parameters():
    print("param {}, grad {}".format(name, param.grad))

which should show None (unless I’m missing how the output is still attached to the computation graph).
Do not call optimizer.zero_grad() before the first backward call as it will fill each .grad attribute with zeros (in the default setup).

5

Thank you, seems that it is better but I still have an issue. The neural network seems to forget to take the second dimension y into account. See the 2 graphs after the codes : the 1st one stands for the nnfunction and the second one is the real function I wish to imitate (hope that you can see the graphs).
Any clue why it still does not work properly ?
Thank you very much for your support.

import numpy as np
import matplotlib.pyplot as plt
import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np

learning_rate = 0.01
num_epochs = 200

N = 200
A = []

U = lambda x: torch.sin(torch.pi * x[0]) * torch.sin(2 * torch.pi * x[1])
f = lambda x, u: (5 * torch.pi**2* torch.sin(torch.pi * x[0]) * torch.sin(2 * torch.pi * x[1]))

x = np.linspace(0, 1, N)
y = np.linspace(0, 1, N)

z = []
for i in range(N):
z.append([x[i],y[i]])
input = torch.tensor(z).to(dtype = torch.float32).requires_grad_(True)

class Net(nn.Module):
def init(self):
super(Net, self).init()
self.fc = nn.Sequential(nn.Linear(2, 64), nn.Tanh(), nn.Linear(64, 1, bias = False))

    self.l1 = nn.Linear(2,128,)
    self.l2 = nn.Linear(128,10)
    self.l3 = nn.Linear(10,1)

def forward(self,input):
    x = input[0]
    y = input[1]
    out = F.tanh(self.l1(input))
    out = F.tanh(self.l2(out))
    out = self.l3(out)
    return out * x * (1-x) * y * (1-y)

model = Net()
gradient = torch.optim.Adam(model.parameters(), lr = learning_rate)

def loss(input, U, A, f):
nablaU = None

for i in input:
    output = model(i)
    #print("output", output)
    Uneuronal_dx = torch.autograd.grad(output, i, grad_outputs = torch.ones_like(output), create_graph=True, retain_graph=True)[0]
    #print("Uneuronal_dx", Uneuronal_dx)
    Uneuronal_d2x = torch.autograd.grad(Uneuronal_dx[0], i, create_graph=True, retain_graph=True)[0]
    #print("Uneuronal_d2x", Uneuronal_d2x)
    
    Uneuronal_dy = torch.autograd.grad(output, i, grad_outputs = torch.ones_like(output), create_graph=True, retain_graph=True)[0]
    #print("Uneuronal_dy", Uneuronal_dy)
    Uneuronal_d2y = torch.autograd.grad(Uneuronal_dy[1], i, create_graph=True, retain_graph=True)[0]
    #print("Uneuronal_d2y", Uneuronal_d2y)

    v = torch.unsqueeze(torch.pow(torch.sub(torch.mul(torch.add(Uneuronal_d2x[0], Uneuronal_d2y[1]), -1),f(i, U)), 2), dim=-1) 
    #print("v", v)
    if nablaU is None:
        nablaU = v
        #print("nablaU", nablaU)
    else:
        nablaU = torch.cat( (nablaU ,v), dim=0)
        #print("nablaU", nablaU)
        
  
return(torch.mean(nablaU))

for i in range(num_epochs):
gradient.zero_grad()
l = loss(input, U, A, f)
l.backward()
print(“loss”, l)
gradient.step()

def u_plot(x, y):
return np.sin(np.pi * x) * np.sin(2 * np.pi * y)

input1 = input[:,0].detach().numpy()
input2 = input[:,1].detach().numpy()

output = []
for i in input:
output.append([model.forward(i).detach().numpy()])

output = np.array(output).reshape(N, 1)

fig = plt.figure()
plt.subplots(figsize=(15, 8))
ax = plt.axes(projection=‘3d’)
X, Y = np.meshgrid(input1, input2)
ax.plot_surface(Y, X, output, rstride=1, cstride=1, edgecolor=‘none’)
ax.set_title(‘Fontion $u(x,y)$’);
ax.set_xlabel(‘y’)
ax.set_ylabel(‘x’)
ax.set_zlabel(‘Network’)

fig = plt.figure()
plt.subplots(figsize=(15, 8))
ax = plt.axes(projection=‘3d’)
X, Y = np.meshgrid(input1, input2)
ax.plot_surface(Y, X, u_plot(X, Y), rstride=1, cstride=1, edgecolor=‘none’)
ax.set_title(‘Fontion $u(x,y)$’);
ax.set_xlabel(‘y’)
ax.set_ylabel(‘x’)
ax.set_zlabel(‘u(x, y)’);

6

image
image816×424 67.8 KB

7

And here is the real function

image
image833×432 65.4 KB

8

Since my last question is more related to the code (and Backward now works), does it make sense that I post a new discussion?

9

Yes, I would suggest to create a new issue regarding details of your model and training procedure.
You’ve also never followed up on my questions and suggestions from my previous post so I don’t know if gradients are calculated at all.

10

OK good to know.
Will do it immediately.
Besides, I have followed up on your suggestions and can tell you that thanks to you, it works!
The code I put here is the code I had issues with the Backward() function.
Now, I am quite near to the initial aim but have still a small other issue :wink:
Thank you very much!