I want to train two agents who play a game (in the game theory sense) through gradient ascent. Each agent has a number of parameters identical to their number of strategies. These parameters are initialized with normally distributed random variables. The parameters get translated into a probability distribution by applying a softmax. I am aware of torch.optim, but want to write the gradient ascent step myself, since I later want to investigate modifications to it. The code below works as expected:
import torch as T
lr = 0.01
std = 0.01
num_strategies1 = 3
num_strategies2 = 5
# Generate random payoff matrices for both players.
payoff_matrix1 = T.randn((num_strategies1, num_strategies2))
payoff_matrix2 = T.randn((num_strategies2, num_strategies1))
params1 = std * T.randn(num_strategies1)
params2 = std * T.randn(num_strategies2)
params1.requires_grad_()
params2.requires_grad_()
for _ in range(10):
probs1 = T.softmax(params1, dim=0)
probs2 = T.softmax(params2, dim=0)
payoff1 = T.dot(probs1, T.matmul(payoff_matrix1, probs2))
payoff2 = T.dot(probs2, T.matmul(payoff_matrix2, probs1))
grad1 = T.autograd.grad(payoff1, params1, create_graph=True)[0]
grad2 = T.autograd.grad(payoff2, params2, create_graph=True)[0]
params1.data += lr * grad1.data # GRADIENT ASCENT
params2.data += lr * grad2.data # GRADIENT ASCENT
Instead of using .data, I want to use the torch.no_grad() context manager. So I replace the two lines marked with GRADIENT ASCENT with the following code:
payoff1.backward(retain_graph=True)
with T.no_grad():
params1 += lr * grad1
params2.grad.zero_()
payoff2.backward()
with T.no_grad():
params2 += lr * grad2
params1.grad.zero_()
params2.grad.zero_()
However, I now get the following error:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [3]] is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
I’ve never had this issue before when using torch.no_grad(). Surprisingly, the issue also disappears if I replace probs1 = T.softmax(params1, dim=0) with a meaningless probs1 = T.sigmoid(params1) (and the same for probs2).
How do I need to change my code such that I can use torch.no_grad()?