Curious about why variable.cuda() is slower than gradient descient

1

Hi,

I am profiling my training code to detect the performance bottleneck. I found that the variable.cuda() operation takes much more time than doing the actual gradient descent(74.1% vs. 13.6%).

Is there any specific reason for this?

Thanks

2

Is there anyone knows?

3

Did you synchronize?

4

No, I didn’t choose any option about synchronize.

5

I mean, you should use torch.cuda.synchronize() to get the “true” time.

6

oh, got it. Thank you!