# Backward() does not work

Dear all,

I have the following code which is used to train a neural function. u(x,y) is my function and I am trying to get it by minimizing the loss (-u’‘(x,y)-f)^2 where u’'(x,y) stands for Laplacian and f as well.
Everything is running fine but I have an issue with the backward() function which does not work here. The loss remains at the exact same level meaning that the neural network does not work.
Can someone help?

import torch
import torch.nn as nn
import numpy as np

learning_rate = 0.01
num_epochs = 15

N = 15

U = lambda x: torch.sin(torch.pi * x[0]) * torch.sin(2 * torch.pi * x[1])
f = lambda x, u: (5 * torch.pi**2* torch.sin(torch.pi * x[0]) * torch.sin(2 * torch.pi * x[1]))

x = np.arange(0, 1.1, 1/(N+1))[1:N+1]
p = np.array([])
for i in range(N):
p = np.append(p, x)
q = np.sort(p)
z = np.column_stack((p, q))

class Net(nn.Module):
def init(self):
super(Net, self).init()
self.fc = nn.Sequential(nn.Linear(2, 64), nn.Tanh(), nn.Linear(64, 1, bias = False))

``````def forward(self, input):
x = input[0]
y = input[1]
input = self.fc(input)
return input * x * (1-x) * y * (1-y)
``````

model = Net()

def loss(x, U, f):
output = model.forward(x)
nabla_x = torch.tensor([0])
nablaU = torch.cat((nablaU , torch.tensor([nabla_x])), dim=0)
return(torch.mean(nablaU[1:]))

for i in range(num_epochs):
l = loss(input, U, f)
l.backward()
print(l)

However, it seems you are detaching tensors implicitly by recreating a tensor in:

``````nabla_x = torch.tensor([torch.sum(nabla_x + grads_inner)])
nablaU = torch.cat((nablaU , torch.tensor([nabla_x])), dim=0)
return(torch.mean(nablaU[1:]))
``````

`nablaU` is now a new tensor, without any Autograd history.
Also, `nablaU` is a newly created leaf tensor also without any Autograd history:

``````nablaU = torch.tensor([0]).to(dtype = torch.float32).requires_grad_(True)
``````

Hi,
Thanks,
Taking into account your comments, it seems to work but I am worried about the loss level.
It has decreased but the level remains high.
It is probably linked to the fact that we are using the second deirvative in the loss. Anyway, I need to recheck it and will revert if need be.
Thank you again for your reactivity

No, I don’t think your approach works as intended.
The `backward` call will not raise an error, since `nablaU` is differentiable, but since it’s a newly created tensor in the `forward` method it will never be updated.
Also, since you are detaching `nabla_x`, the previously used parameters also shouldn’t get any gradients.
Run a single iteration as:

``````output = model(input)
loss = criterion(output, target)
loss.backward()
``````

and check the `.grad` attributes of the model’s parameters via:

``````for name, param in model.named_parameters():
``````

which should show `None` (unless I’m missing how the output is still attached to the computation graph).
Do not call `optimizer.zero_grad()` before the first `backward` call as it will fill each `.grad` attribute with zeros (in the default setup).

Thank you, seems that it is better but I still have an issue. The neural network seems to forget to take the second dimension y into account. See the 2 graphs after the codes : the 1st one stands for the nnfunction and the second one is the real function I wish to imitate (hope that you can see the graphs).
Any clue why it still does not work properly ?
Thank you very much for your support.

import numpy as np
import matplotlib.pyplot as plt
import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np

learning_rate = 0.01
num_epochs = 200

N = 200
A = []

U = lambda x: torch.sin(torch.pi * x[0]) * torch.sin(2 * torch.pi * x[1])
f = lambda x, u: (5 * torch.pi**2* torch.sin(torch.pi * x[0]) * torch.sin(2 * torch.pi * x[1]))

x = np.linspace(0, 1, N)
y = np.linspace(0, 1, N)

z = []
for i in range(N):
z.append([x[i],y[i]])

class Net(nn.Module):
def init(self):
super(Net, self).init()
self.fc = nn.Sequential(nn.Linear(2, 64), nn.Tanh(), nn.Linear(64, 1, bias = False))

``````    self.l1 = nn.Linear(2,128,)
self.l2 = nn.Linear(128,10)
self.l3 = nn.Linear(10,1)

def forward(self,input):
x = input[0]
y = input[1]
out = F.tanh(self.l1(input))
out = F.tanh(self.l2(out))
out = self.l3(out)
return out * x * (1-x) * y * (1-y)
``````

model = Net()

def loss(input, U, A, f):
nablaU = None

``````for i in input:
output = model(i)
#print("output", output)
#print("Uneuronal_dx", Uneuronal_dx)
#print("Uneuronal_d2x", Uneuronal_d2x)

#print("Uneuronal_dy", Uneuronal_dy)
#print("Uneuronal_d2y", Uneuronal_d2y)

v = torch.unsqueeze(torch.pow(torch.sub(torch.mul(torch.add(Uneuronal_d2x[0], Uneuronal_d2y[1]), -1),f(i, U)), 2), dim=-1)
#print("v", v)
if nablaU is None:
nablaU = v
#print("nablaU", nablaU)
else:
nablaU = torch.cat( (nablaU ,v), dim=0)
#print("nablaU", nablaU)

return(torch.mean(nablaU))
``````

for i in range(num_epochs):
l = loss(input, U, A, f)
l.backward()
print(“loss”, l)

def u_plot(x, y):
return np.sin(np.pi * x) * np.sin(2 * np.pi * y)

input1 = input[:,0].detach().numpy()
input2 = input[:,1].detach().numpy()

output = []
for i in input:
output.append([model.forward(i).detach().numpy()])

output = np.array(output).reshape(N, 1)

fig = plt.figure()
plt.subplots(figsize=(15, 8))
ax = plt.axes(projection=‘3d’)
X, Y = np.meshgrid(input1, input2)
ax.plot_surface(Y, X, output, rstride=1, cstride=1, edgecolor=‘none’)
ax.set_title(‘Fontion \$u(x,y)\$’);
ax.set_xlabel(‘y’)
ax.set_ylabel(‘x’)
ax.set_zlabel(‘Network’)

fig = plt.figure()
plt.subplots(figsize=(15, 8))
ax = plt.axes(projection=‘3d’)
X, Y = np.meshgrid(input1, input2)
ax.plot_surface(Y, X, u_plot(X, Y), rstride=1, cstride=1, edgecolor=‘none’)
ax.set_title(‘Fontion \$u(x,y)\$’);
ax.set_xlabel(‘y’)
ax.set_ylabel(‘x’)
ax.set_zlabel(‘u(x, y)’);

And here is the real function

Since my last question is more related to the code (and Backward now works), does it make sense that I post a new discussion?

Yes, I would suggest to create a new issue regarding details of your model and training procedure.
You’ve also never followed up on my questions and suggestions from my previous post so I don’t know if gradients are calculated at all.

OK good to know.
Will do it immediately.
Besides, I have followed up on your suggestions and can tell you that thanks to you, it works!
The code I put here is the code I had issues with the Backward() function.
Now, I am quite near to the initial aim but have still a small other issue
Thank you very much!