What can cause "[WARNING] DeviceCopy in input program"?

compile
1

I have the following warnings:

[2023-03-22 12:32:20,041] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:23,163] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:23,366] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:27,174] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:27,333] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:27,465] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:27,598] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:27,730] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:27,863] torch._inductor.utils: [WARNING] DeviceCopy in input program

I failed to find a good description of what this warning means. Any idea?

1 Like
2

The warning seems to be raised from here and based on the definition of developer_warning I assume it should not be enabled in stable releases. Are you using the latest 2.0.0 release or a nightly build?

3

I’m getting the same warning after installing it through this: PyTorch 2.0 | PyTorch

4

Could you post a minimal and executable code snippet to reproduce the issue, as I’m not seeing this warning using e.g. this simple model:

model = models.resnet50().cuda()
x = torch.randn(1, 3, 224, 224, device="cuda")

model = torch.compile(model)
out = model(x)
5

It seems to be related to the use of Lightning in my case. I tried a minimal snippet, though, with my model, and only got this warning:

[2023-03-23 10:58:49,540] torch._inductor.utils: [WARNING] using triton random, expect difference from eager

6

I’m using a nightly build 2.1.0.dev20230314+cu118. I’ll try with the latest stable release asap.

I only have this warning with one of my models and I couldn’t find which part of the code triggers it. If you have some insight into what could cause this I might be able to provide a minimal executable code snippet.

7

The trigger is likely to be a Device copy which is copying a tensor from one device to another - I would try removing to x_cuda1 = x.to("cuda:1") calls like this in your code and see if it solves the problem

8

That’s right and indeed a simple tensor copy triggers the warning:

def fun(x):
    x = x.to("cuda:0")
    
fun_compiled = torch.compile(fun)

x = torch.randn(1)
out = fun_compiled(x)
# [2023-03-23 10:08:02,025] torch._inductor.utils: [WARNING] DeviceCopy in input program

Did we miss to disable these warning for the stable release?

1 Like
9

Yeah this was a miss - just made the PR Disable inductor developer warnings in official releases by msaroufim · Pull Request #97451 · pytorch/pytorch · GitHub

2 Likes