Sorry to post here, I’m new to this and tried searching all day and can’t quite figure out what to do.
playgame() is a 600-line function that takes a bunch of arguments (valued between 0-1) as “moves” and outputs a score. I have excluded it here, but can pastebin it or something if needed.
class TwoLayerNet(nn.Module):
def __init__(self):
super().__init__()
self.fc1 = nn.Linear(438,100)
self.fc2 = nn.Linear(100,438)
def forward(self, x):
x = F.sigmoid(self.fc1(x))
x = F.sigmoid(self.fc2(x))
return x
m = TwoLayerNet()
loss_fn = nn.MSELoss()
optimizer = optim.Adam(m.parameters(), lr=0.01)
training_epochs = 25
fitnessgoal = torch.tensor(1200, dtype=torch.float)
blanktensor = torch.zeros([1, 438])
def myloss(output, fitnessgoal):
orders = output.tolist()[0]
fitness = torch.tensor(playgame(*orders), dtype=torch.float)
loss = loss_fn(fitness, fitnessgoal)
loss.requires_grad = True
loss.retain_grad()
return loss
for i in range(training_epochs):
# forward pass
fpass = m(blanktensor)
# compute and print loss
loss = myloss(fpass, fitnessgoal)
loss.requires_grad = True
print(i,fitness)
print(loss.grad)
# reset gradients
optimizer.zero_grad()
# backwards pass
loss.backward()
# step the optimizer - update the weights
optimizer.step()
Doing it this way results in the loss remaining exactly the same each epoch. It seems that loss.backward() and optimizer.step() aren’t actually doing anything. loss.grad results in None, which seems to confirm this.
As far as I can gather this is because it can’t back propagate through the playgame() function. I imagine I’m probably severing things when I convert the output from a tensor to a regular list. But playgame() can’t use a tensor as argument, and even if it could, would that even work as a loss function?
I’m trying to train the network to refine a 438 length list of floats (between 0 and 1) that result in the highest possible score when passed through playgame(). Is it possible to do stuff like this with Pytorch?
If you are using just PyTorch functions, Autograd will be able to create the backward pass automatically. However, if you are using plain Python or bumpy methods, you might break the computation graph, which will stop the backward pass.
Which methods are you using, which are not implemented in PyTorch?
Methods? I believe the only kind used in playgame() are .append() for various lists. If you care to look, I’ve pasted the function here: https://pastebin.com/W5anDqcu
Aside from that, you can see that I’m using .tolist() inside the custom loss function
I’m not sure, what the method is doing, but at some point you would have to calculate the loss.
Could you check, which operations are included in the loss calculations?
I removed the myloss() function altogether and fixed the playgame() function so that it can accept a tensor as an argument. The training loop now looks like this:
m = TwoLayerNet()
loss_fn = nn.MSELoss()
optimizer = optim.Adam(m.parameters(), lr=0.01)
training_epochs = 10
fitnessgoal = torch.tensor(1200, dtype=torch.float)
blanktensor = torch.zeros([1, 438])
for i in range(training_epochs):
fpass = m(blanktensor)
loss = loss_fn(playgame(fpass), fitnessgoal)
loss.requires_grad = True
print(i,fitness,loss)
print(loss.grad)
optimizer.zero_grad()
loss.backward()
optimizer.step()
Unfortunately, the result is pretty much the same. 10 epochs:
The loss never changes, and gradient is None. I thought that perhaps the playgame() function was still to blame for this, so I added this to the end of it:
That resulted in the error “you can only change requires_grad flags of leaf variables”. I don’t exactly know what this means, but it seems that running a tensor through the playgame() function is disconnecting things somehow. Maybe Pytorch wasn’t meant to be used like this.
It still seems you are detaching the computation graph somewhere.
If you want to backpropagate through the model, your forward and loss function should use tensors and contain PyTorch methods. Otherwise you would have to implement the backward function manually.
In your updated myloss function, you are re-wrapping the output of playgame in a tensor, which will also break the graph.
That’s from the playgame() function. I did this because otherwise I got an error about integers not having a ‘size’ attribute.
So far I’ve felt pretty sure that it’s the playgame() function detaching things somehow, but I also just tested the forward pass itself. It looks like a pretty standard network architecture, and it’s applying weights to the zerod tensor. But when I print the .grad attribute for its output, that also gets None.
class TwoLayerNet(nn.Module):
def __init__(self):
super().__init__()
self.fc1 = nn.Linear(438,100)
self.fc2 = nn.Linear(100,438)
def forward(self, x):
x = F.sigmoid(self.fc1(x))
x = F.sigmoid(self.fc2(x))
return x
m = TwoLayerNet()
for i in range(training_epochs):
fpass = m(blanktensor)
print(fpass.grad)
So maybe there’s some kind of issue with this as well?