Hi all,
I wonder if anyone knows that TorchRL CQL supports discrete action? I’ve check the documentation, but it is not clear if it supports or not.
Thanks!
Hi @Deep_Gravity
thanks for the question.
We use dist.rsample() which works with continuous values only (except if you’re using tricks to reparametrize discrete values).
This is something we can defo work on for the next version, as the paper tests the results on discrete settings too.
@BY571 any thoughts?
1 Like
Hey @Deep_Gravity,
currently, TorchRL does not support the discrete (DQN-) CQL version. However, implementing it is straightforward, we might have to check how to make it work with the current continuous version or create a standalone discrete CQL loss like the discrete SAC @vmoens . I will give it a shot this week, @Deep_Gravity if you want you can follow the progress here [WIP] Discrete CQL by BY571 · Pull Request #1666 · pytorch/rl · GitHub. It might not get merged quickly but once the objective is done you could probably run it from that branch if you need to.
2 Likes
@Deep_Gravity I added the loss (+ example for cartpole) in the PR and it works already. So in case you need to you can run experiments on that branch if you dont want to wait until the merge.
1 Like
Dear @BY571, thank you very much. I will look into it this week and might have some questions for which I’ll come back to you again 