I work on Flame, an open-source distributed engine for elastic workloads. We added a Search-R1-style training example that teaches Qwen2.5-3B-Instruct to search a BM25 corpus while answering HotpotQA questions.
The training loop runs as Python services: vLLM executors generate trajectories, PyTorch trainer replicas update the policy using group-relative advantages, and Flame’s object cache carries batches and weight updates.
The interesting infrastructure pieces:
- Warm executors: models, retrieval indexes, and optimizer state stay loaded between calls.
- Elastic rollouts: generation capacity can grow with pending work.
- Gang-scheduled trainers: the trainer group receives its minimum resources together, avoiding partially scheduled collective participants.
- Incremental weight sync: after the first full export, existing rollout executors fetch only changed bf16 values.
In a 100-step run using 14 rollout GPUs and two trainer GPUs, held-out exact match improved from 14.5% to 47%, averaging 24.3 seconds per step after the first two steps. Steady-state weight publication was approximately 165 MB per step, versus a 6.17 GB full model.
Caveats: one seed, 200 evaluation questions, and a corpus built from the selected questions’ context paragraphs. These scores aren’t directly comparable with published Search-R1 benchmarks. The synchronous loop also leaves substantial GPU idle time.
For others building agent-training systems: how are you handling weight synchronization and the tradeoff between synchronous updates and policy lag?