The error about needing retain_graph=True is likely unrelated to your symmetric matrix.
I tried the following without getting any error.
T = nn.Parameter(torch.Tensor(5,5))
T.data.normal_()
Q = torch.mm(T, T.t())
out = torch.nn.functional.linear(torch.rand(2,5), Q)
out.mean().backward()
print(T.grad)
If your model is a recurrent model, then you should probably detach the hidden state between batches. See Adding new hidden layer to LSTM
If your model is not a recurrent model, then something else weird is happening, can you provide a small example that produces the error?