What can cause "[WARNING] DeviceCopy in input program"?

I have the following warnings:

[2023-03-22 12:32:20,041] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:23,163] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:23,366] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:27,174] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:27,333] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:27,465] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:27,598] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:27,730] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:27,863] torch._inductor.utils: [WARNING] DeviceCopy in input program

I failed to find a good description of what this warning means. Any idea?

1 Like

The warning seems to be raised from here and based on the definition of developer_warning I assume it should not be enabled in stable releases. Are you using the latest 2.0.0 release or a nightly build?

I’m getting the same warning after installing it through this: PyTorch 2.0 | PyTorch

Could you post a minimal and executable code snippet to reproduce the issue, as I’m not seeing this warning using e.g. this simple model:

model = models.resnet50().cuda()
x = torch.randn(1, 3, 224, 224, device="cuda")

model = torch.compile(model)
out = model(x)

It seems to be related to the use of Lightning in my case. I tried a minimal snippet, though, with my model, and only got this warning:

[2023-03-23 10:58:49,540] torch._inductor.utils: [WARNING] using triton random, expect difference from eager

I’m using a nightly build 2.1.0.dev20230314+cu118. I’ll try with the latest stable release asap.

I only have this warning with one of my models and I couldn’t find which part of the code triggers it. If you have some insight into what could cause this I might be able to provide a minimal executable code snippet.

The trigger is likely to be a Device copy which is copying a tensor from one device to another - I would try removing to x_cuda1 = x.to("cuda:1") calls like this in your code and see if it solves the problem

That’s right and indeed a simple tensor copy triggers the warning:

def fun(x):
    x = x.to("cuda:0")
    
fun_compiled = torch.compile(fun)

x = torch.randn(1)
out = fun_compiled(x)
# [2023-03-23 10:08:02,025] torch._inductor.utils: [WARNING] DeviceCopy in input program

Did we miss to disable these warning for the stable release?

1 Like

Yeah this was a miss - just made the PR Disable inductor developer warnings in official releases by msaroufim · Pull Request #97451 · pytorch/pytorch · GitHub

2 Likes

What does “input program” mean in this context?
For stable diffusion of course there is a transfer of the image off of the GPU after the VAE/decode_first_stage step. Also, I suspect the initial noise is transferred to the GPU at the beginning. It’d be nice to know where in the complex set of operations that Stable Diffusion is, where this is coming from.

I have torch 2.0.1+cu118 and trying the default ‘inductor’.