What can cause "[WARNING] DeviceCopy in input program"?

torch.compile
1

I have the following warnings:

[2023-03-22 12:32:20,041] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:23,163] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:23,366] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:27,174] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:27,333] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:27,465] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:27,598] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:27,730] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:27,863] torch._inductor.utils: [WARNING] DeviceCopy in input program

I failed to find a good description of what this warning means. Any idea?

1 Like
2

The warning seems to be raised from here and based on the definition of developer_warning I assume it should not be enabled in stable releases. Are you using the latest 2.0.0 release or a nightly build?

3

Iā€™m getting the same warning after installing it through this: PyTorch 2.0 | PyTorch

4

Could you post a minimal and executable code snippet to reproduce the issue, as Iā€™m not seeing this warning using e.g. this simple model:

model = models.resnet50().cuda()
x = torch.randn(1, 3, 224, 224, device="cuda")

model = torch.compile(model)
out = model(x)
5

It seems to be related to the use of Lightning in my case. I tried a minimal snippet, though, with my model, and only got this warning:

[2023-03-23 10:58:49,540] torch._inductor.utils: [WARNING] using triton random, expect difference from eager

6

Iā€™m using a nightly build 2.1.0.dev20230314+cu118. Iā€™ll try with the latest stable release asap.

I only have this warning with one of my models and I couldnā€™t find which part of the code triggers it. If you have some insight into what could cause this I might be able to provide a minimal executable code snippet.

7

The trigger is likely to be a Device copy which is copying a tensor from one device to another - I would try removing to x_cuda1 = x.to("cuda:1") calls like this in your code and see if it solves the problem

8

Thatā€™s right and indeed a simple tensor copy triggers the warning:

def fun(x):
    x = x.to("cuda:0")
    
fun_compiled = torch.compile(fun)

x = torch.randn(1)
out = fun_compiled(x)
# [2023-03-23 10:08:02,025] torch._inductor.utils: [WARNING] DeviceCopy in input program

Did we miss to disable these warning for the stable release?

1 Like
9

Yeah this was a miss - just made the PR Disable inductor developer warnings in official releases by msaroufim Ā· Pull Request #97451 Ā· pytorch/pytorch Ā· GitHub

2 Likes
10

What does ā€œinput programā€ mean in this context?
For stable diffusion of course there is a transfer of the image off of the GPU after the VAE/decode_first_stage step. Also, I suspect the initial noise is transferred to the GPU at the beginning. Itā€™d be nice to know where in the complex set of operations that Stable Diffusion is, where this is coming from.

I have torch 2.0.1+cu118 and trying the default ā€˜inductorā€™.