@ptrblck Thank you for your response. As suggested, I have taken 10 samples and tried changing hyperparameters multiple times. I have tried different loss functions, learning rates, removing the initialization of parameters and so on. Also require_grad has been removed. But I am getting the same output for all the inputs. I am using the below code to get the output and comparing with the target manually by plotting.
index=38
with torch.no_grad():
y_pred=net(X_val[index].view(1,1,2048).float())