Troubleshooting Parallel Training and Validation on the Same GPU with Multiple Threads

If you are using Python’s multi-threading you should be aware of the GIL and its limitations.
Besides that your device would not only need to provide enough memory but also compute resources. Details including a great GTC talk can be found here.