Per the PR that introduced this splitting, Improving memory utilization of Z2+MoE by siddharth9820 · Pull Request #2079 · microsoft/DeepSpeed · GitHub , it is to save memory because only one of these groups will be on the GPU in full precision at a time.
Please take the answer with a grain of salt, the question just piqued my curiosity.
Best regards
Thomas