Hybrid Execution

Is it possible to do computation simultaneously on host cpu device and gpu device ?

You mean, training a model on the CPU and the GPU simultaneously? I can’t think of any reason you would want to do that because the CPU is so much slower that anything it gets would be computed faster by being queued for the GPU.