Hello there,
I’m trying to implement (copypaste from different sources) the DDQN algorithm in PyTorch. I get the following error when calling loss.backward(): RuntimeError: Function MmBackward returned an invalid gradient at index 1 - got [128, 16] but expected shape compatible with [4, 16].
I believe this is the relevant code:
transitions = self.memory.sample()
batch_state, batch_next_state, batch_action, batch_reward = zip(*transitions)
batch_state = torch.cat(batch_state)
batch_next_state = torch.cat(batch_next_state)
batch_action = torch.cat(batch_action).squeeze(1)
batch_reward = torch.cat(batch_reward)
# current Q values are estimated by NN for all actions
current_q_values = self.policy_net(batch_state).gather(0, batch_action)
# expected Q values are estimated from actions which gives maximum Q value
max_next_q_values, _ = self.target_net(batch_next_state).detach().max(0)
expected_q_values = batch_reward + (self.gamma * max_next_q_values)
# Compute loss
loss = self.criterion(current_q_values, expected_q_values)
# Zero the gradients and backpropagate
self.optimizer.zero_grad()
loss.backward()
for param in self.policy_net.parameters():
param.grad.data.clamp_(-1, 1)
# Optimizer
self.optimizer.step()
Both policy_net and target_net are just two linear layers with ReLU activation.
The error makes me believe that the graph is messed up somewhere, but I’m not sure how such a situation can be reached. I assume MmBackward refers to a torch.mm computation, but I don’t think I ever call torch.mm.
I am incredible lost and any help would be much appreciated. Thanks in advance!