---
title: "Reinforcement learning is an infrastructure problem"
description: "Reinforcement learning for post-training large language models has become infrastructure-constrained as teams move to multi-node configurations for larger models like Qwen 3 30B and Kimi K2.6. The RL loop divides into three independent infrastructure challenges: training with reliable forward and backward passes, high-performance inference rollouts, and thousands of concurrent sandbox environments for policy execution. Weight synchronization between trainer and rollout engine becomes the critical bottleneck in multi-node setups, with RDMA offering up to 100x speedup over TCP and significant cost savings per step. Teams commonly get stuck maintaining glue code, queueing for cluster time, and under-utilizing GPUs when sandbox buffers are not correctly sized. Modal addresses these with RDMA-connected clusters via a single rdma=True keyword, fast container boot, and the new open-source Modal Training Gym library that abstracts common scaffolding into under 100 lines of code."
---

# Reinforcement learning is an infrastructure problem

[Modal](https://yomu.fyi/company/modal) · Jun 1, 2026

**Type:** Announcement

## Summary

Reinforcement learning for post-training large language models has become infrastructure-constrained as teams move to multi-node configurations for larger models like Qwen 3 30B and Kimi K2.6. The RL loop divides into three independent infrastructure challenges: training with reliable forward and backward passes, high-performance inference rollouts, and thousands of concurrent sandbox environments for policy execution. Weight synchronization between trainer and rollout engine becomes the critical bottleneck in multi-node setups, with RDMA offering up to 100x speedup over TCP and significant cost savings per step. Teams commonly get stuck maintaining glue code, queueing for cluster time, and under-utilizing GPUs when sandbox buffers are not correctly sized. Modal addresses these with RDMA-connected clusters via a single rdma=True keyword, fast container boot, and the new open-source Modal Training Gym library that abstracts common scaffolding into under 100 lines of code.

## Context

The increasing adoption of reinforcement learning for post-training large language models has shifted the bottleneck from algorithms to infrastructure, especially as teams scale to multi-node clusters and larger models.

## Approach / What changed

Modal provides a platform that bundles infrastructure with code, offering RDMA-connected clusters via a single rdma=True keyword, fast container boot, and the new open-source Modal Training Gym library that abstracts common scaffolding into under 100 lines of code.

## Takeaways

- The RL post-training loop consists of three independent infrastructure challenges: a training engine for forward/backward passes, high-performance inference for rollouts, and isolated environments for concurrent policy execution.
- Weight synchronization between the trainer and rollout engine is the critical bottleneck in multi-node training; RDMA provides up to 100x speedup over TCP, with LoRA and delta compression reducing transfer sizes.
- Teams commonly fail due to glue code maintenance, queueing for cluster time, and GPU under-utilization from improperly sized sandbox buffers; Modal’s abstractions like rdma=True and fast sandbox startup address these.

## FAQ

### How does RDMA improve training speed compared to TCP?

RDMA transfer speed improves training by up to 100x compared to TCP within the same cluster. For example, transferring a 16.3 GB weight update for Qwen3 8B takes 2.62 seconds over TCP but only 41 milliseconds over RDMA, saving an estimated $0.21 per training step.

### What are the common problems teams face when scaling RL?

Teams get stuck on three problems: maintaining glue code for node management, queueing for cluster time due to compute shortages, and under-utilizing GPUs when sandbox buffers are not properly sized to prevent blocking the GPU on environment spin-up.

### What is the Modal Training Gym?

The Modal Training Gym is an experimental open-source library that abstracts common RL training scaffolding into under 100 lines of code. Users define a reward function, model, and environment, and the library handles multi-node clusters, environment orchestration, and observability.

**Tags:** [GPU](https://yomu.fyi/topic/gpu), [LLMs](https://yomu.fyi/topic/llm), [Open Source](https://yomu.fyi/topic/open-source), [Reinforcement Learning](https://yomu.fyi/topic/reinforcement-learning)

- Source: [Modal](https://modal.com/blog/reinforcement-learning-infrastructure-problem)
- Source URL: https://modal.com/blog/reinforcement-learning-infrastructure-problem
- Ingested by Yomu: 2026-09-20T07:46:54.124Z

[Read original post](https://modal.com/blog/reinforcement-learning-infrastructure-problem)