# reinforcement-learning **URL:** https://discuss.pytorch.org/c/reinforcement-learning/6.md [Latest](https://discuss.pytorch.org/latest.md) · [Categories](https://discuss.pytorch.org/categories.md) --- ## [About the reinforcement-learning category](https://discuss.pytorch.org/t/about-the-reinforcement-learning-category/23)
**Author:** [@smth](https://discuss.pytorch.org/u/smth)\ **Replies:** 7\ **Last updated:** [October 18, 2023, 1:35pm UTC](https://discuss.pytorch.org/t/about-the-reinforcement-learning-category/23 "2023-10-18T13:35:12Z")
A section to discuss RL implementations, research, problems --- ## [Training a Search-R1 agent with Flame: warm GPU services, gang scheduling, and incremental weight sync](https://discuss.pytorch.org/t/training-a-search-r1-agent-with-flame-warm-gpu-services-gang-scheduling-and-incremental-weight-sync/225581)
**Author:** [@k82cn](https://discuss.pytorch.org/u/k82cn)\ **Replies:** 0\ **Last updated:** [October 20, 2026, 5:28am UTC](https://discuss.pytorch.org/t/training-a-search-r1-agent-with-flame-warm-gpu-services-gang-scheduling-and-incremental-weight-sync/225581 "2026-10-20T05:28:58Z")
I work on Flame, an open-source distributed engine for elastic workloads. We added a Search-R1-style training example that teaches Qwen2.5-3B-Instruct to search a BM25 corpus while answering HotpotQA questions. The trai… --- ## [Reminder-dsl-mcp](https://discuss.pytorch.org/t/reminder-dsl-mcp/225104)
**Author:** [@Anderson\_Estiver\_Alv](https://discuss.pytorch.org/u/Anderson_Estiver_Alv)\ **Replies:** 0\ **Last updated:** [June 16, 2026, 11:08am UTC](https://discuss.pytorch.org/t/reminder-dsl-mcp/225104 "2026-06-16T11:08:18Z")
from sklearn.metrics import f1\_score, make\_scorer from sklearn.model\_selection import GridSearchCV, StratifiedKFold param\_grid = \[ { ‘est\_\_C’: \[0.5, 0.75, 1, 1.5, 2, 3\], ‘est\_\_kernel’: \[‘rbf’\], # others does not wor… --- ## [Bridging the Gap: Using PyTorch for Intelligent Agent Design in Game Environments](https://discuss.pytorch.org/t/bridging-the-gap-using-pytorch-for-intelligent-agent-design-in-game-environments/224642)
**Author:** [@aadarshkumar\_edu](https://discuss.pytorch.org/u/aadarshkumar_edu)\ **Replies:** 0\ **Last updated:** [March 2, 2026, 1:09pm UTC](https://discuss.pytorch.org/t/bridging-the-gap-using-pytorch-for-intelligent-agent-design-in-game-environments/224642 "2026-03-02T13:09:50Z")
Hi everyone, I’ve been exploring the intersection of deep learning and real-time game environments. While PyTorch is the de-facto standard for research, I’ve noticed a significant “implementation gap” when it comes to b… --- ## [How to instantiate double LSTM + MLP for actor/critic in PPO](https://discuss.pytorch.org/t/how-to-instantiate-double-lstm-mlp-for-actor-critic-in-ppo/224240)
**Author:** [@yoan](https://discuss.pytorch.org/u/yoan)\ **Replies:** 0\ **Last updated:** [December 20, 2025, 2:49pm UTC](https://discuss.pytorch.org/t/how-to-instantiate-double-lstm-mlp-for-actor-critic-in-ppo/224240 "2025-12-20T14:49:09Z")
Hi ! I am trying to write a model for lunar lander v3 using PPO using double LSTM +MLP, one for actor and one for critic. I successfully implemented the actor but struggling for the critic. I am puzzled as it should not… --- ## [Performance Differences in TD3 Training When Switching from NumPy 1.26.0 to NumPy 2.2.6](https://discuss.pytorch.org/t/performance-differences-in-td3-training-when-switching-from-numpy-1-26-0-to-numpy-2-2-6/224135)
**Author:** [@chouakifares](https://discuss.pytorch.org/u/chouakifares)\ **Replies:** 0\ **Last updated:** [December 9, 2025, 8:16am UTC](https://discuss.pytorch.org/t/performance-differences-in-td3-training-when-switching-from-numpy-1-26-0-to-numpy-2-2-6/224135 "2025-12-09T08:16:58Z")
Hi everyone, I’m currently training a TD3 agent using Stable-Baselines3 and I’ve run into a surprising issue related to NumPy versions. After a lot of debugging, I’ve noticed that my training results differ significantl… --- ## [Several implementations of TruncatedNormal?](https://discuss.pytorch.org/t/several-implementations-of-truncatednormal/224071)
**Author:** [@adrische](https://discuss.pytorch.org/u/adrische)\ **Replies:** 0\ **Last updated:** [December 1, 2025, 4:23pm UTC](https://discuss.pytorch.org/t/several-implementations-of-truncatednormal/224071 "2025-12-01T16:23:10Z")
Hi, There seem to be two implementations of TruncatedNormal, Both are similar, but they are not the same. Why is this, and which should I use? Thank you! --- ## [PPO learning poorly on LunarLander-v3](https://discuss.pytorch.org/t/ppo-learning-poorly-on-lunarlander-v3/223951)
**Author:** [@yoan](https://discuss.pytorch.org/u/yoan)\ **Replies:** 2\ **Last updated:** [November 18, 2025, 7:39am UTC](https://discuss.pytorch.org/t/ppo-learning-poorly-on-lunarlander-v3/223951 "2025-11-18T07:39:04Z")
Hi! I am using PPO to try to learn LunarLander-v3 in torchrl. I am having good results (avg reward \> 280) using stable-baselines3 (sb3) with less than 1M steps. The thing is, I have tried a bunch of different things lis… --- ## [Apparent RAM memory leak when converting batch of ndarray states to GPU tensor](https://discuss.pytorch.org/t/apparent-ram-memory-leak-when-converting-batch-of-ndarray-states-to-gpu-tensor/223740)
**Author:** [@samnz](https://discuss.pytorch.org/u/samnz)\ **Replies:** 0\ **Last updated:** [October 29, 2025, 2:31am UTC](https://discuss.pytorch.org/t/apparent-ram-memory-leak-when-converting-batch-of-ndarray-states-to-gpu-tensor/223740 "2025-10-29T02:31:48Z")
Hello everyone, I have been running into an inconsistent issue when I am attempting to convert a list/batch of states (dictionaries mapping to ndarrays) to tensors on the GPU. For many approaches I have tried, the progr… --- ## [Training Machine Learning Model In Browser For Reinforcement Learning](https://discuss.pytorch.org/t/training-machine-learning-model-in-browser-for-reinforcement-learning/218074)
**Author:** [@Sliferslacker](https://discuss.pytorch.org/u/Sliferslacker)\ **Replies:** 1\ **Last updated:** [October 27, 2025, 12:41am UTC](https://discuss.pytorch.org/t/training-machine-learning-model-in-browser-for-reinforcement-learning/218074 "2025-10-27T00:41:16Z")
I’m trying to make a game website and add some reinforcement learning into the games. So my backend for making the website is Django, and I’m currently using PyTorch for the neural network. I’ve made the game on the web … --- ## [Model Boilerplate for a Simple DQN](https://discuss.pytorch.org/t/model-boilerplate-for-a-simple-dqn/223565)
**Author:** [@icarosadero](https://discuss.pytorch.org/u/icarosadero)\ **Replies:** 3\ **Last updated:** [October 22, 2025, 6:07am UTC](https://discuss.pytorch.org/t/model-boilerplate-for-a-simple-dqn/223565 "2025-10-22T06:07:45Z")
Could someone please provide me a simple, bare-bones DQN training loop? I have looked at the examples for the \[TorchRL documentation\]( TorchRL — torchrl 0.0 documentation ), but it becomes overwhelmingly complex, and aft… --- ## [Implementation of Hierarchical Actor Critic with PPolicy-on Policy-off Policy Optimization for primitive actions](https://discuss.pytorch.org/t/implementation-of-hierarchical-actor-critic-with-ppolicy-on-policy-off-policy-optimization-for-primitive-actions/223612)
**Author:** [@Lordking1624](https://discuss.pytorch.org/u/Lordking1624)\ **Replies:** 0\ **Last updated:** [October 15, 2025, 8:51am UTC](https://discuss.pytorch.org/t/implementation-of-hierarchical-actor-critic-with-ppolicy-on-policy-off-policy-optimization-for-primitive-actions/223612 "2025-10-15T08:51:35Z")
Hello everyone, I’m currently undertaking a Hierarchical Reinforcement Learning (HRL) project and I am attempting to implement a hybrid architecture based on the Hierarchical Actor-Critic (HAC) framework. I started fro… --- ## [PyTorch Compatibility with Older CUDA Versions](https://discuss.pytorch.org/t/pytorch-compatibility-with-older-cuda-versions/223220)
**Author:** [@Harishankar14](https://discuss.pytorch.org/u/Harishankar14)\ **Replies:** 1\ **Last updated:** [September 20, 2025, 3:46pm UTC](https://discuss.pytorch.org/t/pytorch-compatibility-with-older-cuda-versions/223220 "2025-09-20T15:46:54Z")
Hey Guys, I have been working on an ML project, so i have a GPU server (An ancient one) which is backed by CUDA 3.0 .So what is the minimum version supported for PyTorch ? Thank You :slight\_smile: --- ## [Agent Masking in Multi-agent environment?](https://discuss.pytorch.org/t/agent-masking-in-multi-agent-environment/222925)
**Author:** [@Jean-Marc\_Fauche](https://discuss.pytorch.org/u/Jean-Marc_Fauche)\ **Replies:** 0\ **Last updated:** [September 7, 2025, 2:32pm UTC](https://discuss.pytorch.org/t/agent-masking-in-multi-agent-environment/222925 "2025-09-07T14:32:32Z")
Hi, I would like to know if there is a way to implement an “agent mask”. To clarify things: let’s imagine an environment that for certain steps provides data or not to the agents. Some agents have it, others do not. It w… --- ## [Batching a multicategorical spec](https://discuss.pytorch.org/t/batching-a-multicategorical-spec/221780)
**Author:** [@rsarpongstreetor](https://discuss.pytorch.org/u/rsarpongstreetor)\ **Replies:** 4\ **Last updated:** [August 27, 2025, 7:41am UTC](https://discuss.pytorch.org/t/batching-a-multicategorical-spec/221780 "2025-08-27T07:41:52Z")
Can I batch a multicategoral spec as shown below. How do i it . I have tried inheriting the composite batch size but its not working. I need it to work in my MARL environment . if self.categorical\_actions: self.actio… --- ## [How to pass options to env.reset within a data collector](https://discuss.pytorch.org/t/how-to-pass-options-to-env-reset-within-a-data-collector/222663)
**Author:** [@icarosadero](https://discuss.pytorch.org/u/icarosadero)\ **Replies:** 0\ **Last updated:** [August 26, 2025, 12:06pm UTC](https://discuss.pytorch.org/t/how-to-pass-options-to-env-reset-within-a-data-collector/222663 "2025-08-26T12:06:21Z")
If you create a \`GymEnv\` you can pass keyword arguments to the reset method in the usual way: env.reset(options={"custom\_arg":"custom\_value"}) I would like to know how to do the same for environments instantiated by d… --- ## [Environments from scratch with Torchrl](https://discuss.pytorch.org/t/environments-from-scratch-with-torchrl/189767)
**Author:** [@Jean-Marc\_Fauche](https://discuss.pytorch.org/u/Jean-Marc_Fauche)\ **Replies:** 17\ **Last updated:** [August 25, 2025, 6:39am UTC](https://discuss.pytorch.org/t/environments-from-scratch-with-torchrl/189767 "2025-08-25T06:39:27Z")
Hello, I open this topic … for my contribution here is the simplest one you can imagine! from typing import Optional import matplotlib.pyplot as plt import torch from tensordict.nn import TensorDictModule from tensordi… --- ## [How to manage done in a batched custom Env?](https://discuss.pytorch.org/t/how-to-manage-done-in-a-batched-custom-env/222529)
**Author:** [@Jean-Marc\_Fauche](https://discuss.pytorch.org/u/Jean-Marc_Fauche)\ **Replies:** 3\ **Last updated:** [August 25, 2025, 6:37am UTC](https://discuss.pytorch.org/t/how-to-manage-done-in-a-batched-custom-env/222529 "2025-08-25T06:37:07Z")
Hi, In a batched Env, we have different trajectories. So at the end of step,for a batch\_size of 3 we could have done = False,False,True. But in this case, the \_reset function received a tensordict with only a ‘\_reset’ ke… --- ## [ClipPPOLoss problem with MaskedCategorical dist](https://discuss.pytorch.org/t/clipppoloss-problem-with-maskedcategorical-dist/222394)
**Author:** [@Jean-Marc\_Fauche](https://discuss.pytorch.org/u/Jean-Marc_Fauche)\ **Replies:** 2\ **Last updated:** [August 21, 2025, 12:03am UTC](https://discuss.pytorch.org/t/clipppoloss-problem-with-maskedcategorical-dist/222394 "2025-08-21T00:03:48Z")
Here is a loss module: loss\_module = ClipPPOLoss( actor\_network = policy\_module, critic\_network = critic, clip\_epsilon = clip\_epsilon, entropy\_bonus = bool(entropy\_eps), entropy\_coeff = entropy\_eps, … --- ## [CosTrader Env from scratch... and transform problem](https://discuss.pytorch.org/t/costrader-env-from-scratch-and-transform-problem/222344)
**Author:** [@Jean-Marc\_Fauche](https://discuss.pytorch.org/u/Jean-Marc_Fauche)\ **Replies:** 3\ **Last updated:** [August 15, 2025, 2:23am UTC](https://discuss.pytorch.org/t/costrader-env-from-scratch-and-transform-problem/222344 "2025-08-15T02:23:30Z")
Hello Here is a custom Env. I would like to discussmy mystakes because it pass “check\_env\_specs” (see Tests at the bottom of the code) but not after transformations… CosTrader.py import torch from tensordict.tensordict… --- ## [PPO with Categorical Action... help](https://discuss.pytorch.org/t/ppo-with-categorical-action-help/222172)
**Author:** [@Jean-Marc\_Fauche](https://discuss.pytorch.org/u/Jean-Marc_Fauche)\ **Replies:** 10\ **Last updated:** [August 14, 2025, 7:47pm UTC](https://discuss.pytorch.org/t/ppo-with-categorical-action-help/222172 "2025-08-14T19:47:28Z")
Hello, I’m looking for examples or tutorials on PPO using torchrl with Categorical Actions for a custom Env. I’ve tried to adapt the inverted-double-pendulum tutorial but without success. If anyone can help me I would … --- ## [Question about TorchRL ParallelEnv error on single-gpu device](https://discuss.pytorch.org/t/question-about-torchrl-parallelenv-error-on-single-gpu-device/222004)
**Author:** [@Ivar\_Gaitan](https://discuss.pytorch.org/u/Ivar_Gaitan)\ **Replies:** 3\ **Last updated:** [August 5, 2025, 7:28am UTC](https://discuss.pytorch.org/t/question-about-torchrl-parallelenv-error-on-single-gpu-device/222004 "2025-08-05T07:28:51Z")
Hi! When creating a simple ParallelEnv on a single CUDA device, I get a runtime error. While it may not be a recommended practice, I don’t understand why it fails. Here’s the code (modified from the TorchRL tutorials): i… --- ## [Help understanding data collectors](https://discuss.pytorch.org/t/help-understanding-data-collectors/221438)
**Author:** [@icarosadero](https://discuss.pytorch.org/u/icarosadero)\ **Replies:** 1\ **Last updated:** [August 4, 2025, 9:38pm UTC](https://discuss.pytorch.org/t/help-understanding-data-collectors/221438 "2025-08-04T21:38:41Z")
I am following the RNN tutorial for reinforcement learning in PyTorch from this link. In there, I came across the SyncDataCollector class, whose documentation looked a bit confusing. I have the following questions: Wha… --- ## [Should we split the trajectories prior to calculating the loss for a DQN?](https://discuss.pytorch.org/t/should-we-split-the-trajectories-prior-to-calculating-the-loss-for-a-dqn/221749)
**Author:** [@icarosadero](https://discuss.pytorch.org/u/icarosadero)\ **Replies:** 1\ **Last updated:** [August 4, 2025, 9:31pm UTC](https://discuss.pytorch.org/t/should-we-split-the-trajectories-prior-to-calculating-the-loss-for-a-dqn/221749 "2025-08-04T21:31:18Z")
Consider this minimal example where data is sampled from a collector according to a MLP policy: from torchrl.collectors import SyncDataCollector from torchrl.modules import MLP, QValueModule from torchrl.collectors.util… --- ## [Question About If PPO Training Will Work](https://discuss.pytorch.org/t/question-about-if-ppo-training-will-work/221899)
**Author:** [@Rohan\_Jay](https://discuss.pytorch.org/u/Rohan_Jay)\ **Replies:** 1\ **Last updated:** [July 29, 2025, 6:37pm UTC](https://discuss.pytorch.org/t/question-about-if-ppo-training-will-work/221899 "2025-07-29T18:37:02Z")
Hi. I am relatively inexperienced in PyTorch so I apologize if this question seems silly. I am working with some friends on implementing PPO from scratch using PyTorch, and part of what I have to do is training the acto… --- ## [RTX 5090 interconnection with pytorch](https://discuss.pytorch.org/t/rtx-5090-interconnection-with-pytorch/221631)
**Author:** [@AliNoutash](https://discuss.pytorch.org/u/AliNoutash)\ **Replies:** 6\ **Last updated:** [July 28, 2025, 1:49pm UTC](https://discuss.pytorch.org/t/rtx-5090-interconnection-with-pytorch/221631 "2025-07-28T13:49:22Z")
Dear all I have installed NVIDIA RTX 5090 on my PC for deep rienforcement leerning with pycharm. My PC RAM is 16 GB, and CPU is 12th Gen Intel(R) Core™ i7-12700. However, when I run my code the system freezes and goes t… --- ## [Model almost instantly produces "nan"](https://discuss.pytorch.org/t/model-almost-instantly-produces-nan/221483)
**Author:** [@Jake\_Watson](https://discuss.pytorch.org/u/Jake_Watson)\ **Replies:** 4\ **Last updated:** [July 19, 2025, 8:44pm UTC](https://discuss.pytorch.org/t/model-almost-instantly-produces-nan/221483 "2025-07-19T20:44:56Z")
Hello all, I’ve been attempting to create a simple implementation of the A2C algorithm to play the Atari games. While at first I had some success (model would train but performance was poor), I decided to make some modi… --- ## [What loss function should the inner loop of MAML use?](https://discuss.pytorch.org/t/what-loss-function-should-the-inner-loop-of-maml-use/214371)
**Author:** [@Vulcan88](https://discuss.pytorch.org/u/Vulcan88)\ **Replies:** 2\ **Last updated:** [June 27, 2025, 6:47pm UTC](https://discuss.pytorch.org/t/what-loss-function-should-the-inner-loop-of-maml-use/214371 "2025-06-27T18:47:36Z")
Hi, I’m implementing MAML in the Reinforcement Learning domain. I’m planning to use PPO training for validation. However I want to know if the PPO loss function is appropriate during Meta-Training. The emphasis on gradu… --- ## [TruncatedNormal loc argument](https://discuss.pytorch.org/t/truncatednormal-loc-argument/220898)
**Author:** [@alicemabille](https://discuss.pytorch.org/u/alicemabille)\ **Replies:** 3\ **Last updated:** [June 19, 2025, 7:24am UTC](https://discuss.pytorch.org/t/truncatednormal-loc-argument/220898 "2025-06-19T07:24:25Z")
I just discovered the truncated normal distribution and I’m trying to use torchrl TruncatedNormal but I can’t find anythin about “location” in the paper that introduced this idea and I don’t understand torchrl’s document… --- ## [Using buffers in ParallelEnvs / MultiSyncCollectors](https://discuss.pytorch.org/t/using-buffers-in-parallelenvs-multisynccollectors/217363)
**Author:** [@wlruys](https://discuss.pytorch.org/u/wlruys)\ **Replies:** 2\ **Last updated:** [June 16, 2025, 5:30pm UTC](https://discuss.pytorch.org/t/using-buffers-in-parallelenvs-multisynccollectors/217363 "2025-06-16T17:30:09Z")
Hi, I’ve been moving a custom environment from dynamically sized to statically known observation sizes and I came across unexpected behavior in how the input and output TensorDicts in \_step() are behaving. What is the e… [Next page](https://discuss.pytorch.org/c/reinforcement-learning/6.md?page=1)