Regarding using different GPUs, the model effect drops off a cliff.

My result on the training set on Nvidia 1060 was 97%, and the result on the test set was 94%. But after I switched to Nvidia V100, the result of my training set was 100%, and the result of the test set was only 30%. I was very confused about what was causing it.

I would recommend checking the datasets and actual training script as we have seen a few issues in this discussion board claiming a different GPU causes convergence issues while it was related to something else. Just recently this post claimed the same and it turned out some dataset folders were empty.