If you don’t use to() to push your data and models to a GPU, all tensors will be created on the CPU by default.
Setting CUDA_VISIBLE_DEVICES won’t change this behavior, so you would still need to somehow declare which tensor should be pushed to the GPU and which should stay on the CPU.
You could try to set the default tensor type to a CUDA one, if a GPU was found, but that feels like an unnecessary hack, since now all calculations will be performed on the GPU by default.
Asynchronous CUDA calls with some CPU computation will hardly be possible.
Could you explain your idea a bit more, as I might misunderstand it?