During reinforcement learning (RL) post-training, training trajectories are generated on the fly by running inference with the current policy model. For training efficiency, it is common to run inference and training in parallel on separate GPU resources in a disaggregated setup. Although the policy model is updated only on the training GPUs, the system must periodically refresh the weights on the inference GPUs with the latest parameters generated on the training GPUs. This transfer of model weights between training and inference GPUs is called refit.
Refit involves two defining characteristics that pose challenges. First, inference and training have different computational requirements, so the weights are distributed differently across inference and training instances by having different parallel mappings (data, context, tensor, expert, and pipeline parallelism).
Second, because refit must update all model weights on the inference GPUs, the communication payload is on the order of the total size of the model’s weights. This payload is already very large for the latest frontier models and is expected to continue growing. Together, these characteristics mean that refit requires high-performance communication across a wide variety of distributed source and destination layouts.
This post explains how AI training frameworks can address these refit challenges using the Reshard collective, a new type of collective communication primarily designed to accelerate RL weight refit. NCCL M2N Reshard, an…
Source: NVIDIA Technical Blog — Read the full article on the original site ↗