I don’t think it’s safe/supported to reset CUDA_VISIBLE_DEVICES in the middle of script execution (e.g., after CUDA has been initialized): Os.environ ["CUDA_VISIBLE_DEVICES"] not functioning - #4 by albanD
If you want to restrict the number of GPUs deepspeed is using, I would check if e.g., setting the world_size parameter or something like nproc_per_node could achieve this.