Hey xuehy. Sorry since it was not very successful so I didn’t keep a lot of records. What I was doing was just adjusting hyperparameters. What I do remember is, memory capacity matters a lot.
So far I haven’t found a solution to make DQN converge. The potential reasons in my mind are 1) NN models are prone to divergence in theory; 2) simple image recognition model doesn’t learn CartPole very well.
I don’t know how far did you get since then, but I would suggest both include a LSTM cell, and try your model on other games. I haven’t done them yet though.