I got a new gpu and installed the newest nightly build in order to use cuda 12.8.
On an older build of torch I never got this error, but with the same exact code on the newer build I get this:
E0423 07:47:56.453000 2930 torch/subclasses/fake_tensor.py:2613] [1/1] failed while attempting to run meta for aten.set.source_Storage
E0423 07:47:56.453000 2930 torch/_subclasses/fake_tensor.py:2613] [1/1] Traceback (most recent call last):
E0423 07:47:56.453000 2930 torch/_subclasses/fake_tensor.py:2613] [1/1] File “/home/jason/myai2/lib/python3.12/site-packages/torch/_subclasses/fake_tensor.py”, line 2609, in _dispatch_impl
E0423 07:47:56.453000 2930 torch/_subclasses/fake_tensor.py:2613] [1/1] r = func(*args, **kwargs)
E0423 07:47:56.453000 2930 torch/_subclasses/fake_tensor.py:2613] [1/1] ^^^^^^^^^^^^^^^^^^^^^
E0423 07:47:56.453000 2930 torch/_subclasses/fake_tensor.py:2613] [1/1] File “/home/jason/myai2/lib/python3.12/site-packages/torch/_ops.py”, line 806, in call
E0423 07:47:56.453000 2930 torch/_subclasses/fake_tensor.py:2613] [1/1] return self._op(*args, **kwargs)
E0423 07:47:56.453000 2930 torch/_subclasses/fake_tensor.py:2613] [1/1] ^^^^^^^^^^^^^^^^^^^^^^^^^
E0423 07:47:56.453000 2930 torch/_subclasses/fake_tensor.py:2613] [1/1] RuntimeError: Attempted to set the storage of a tensor on device “meta” to a storage on different device “cuda:0”. This is no longer allowed; the devices must match.
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] failed to eagerly compile backwards for dynamic, suppressing in case backwards not needed
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] Traceback (most recent call last):
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] File “/home/jason/myai2/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py”, line 1222, in aot_dispatch_autograd
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] copy.deepcopy(bw_module), placeholder_list
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] ^^^^^^^^^^^^^^^^^^^^^^^^
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] File “/usr/lib/python3.12/copy.py”, line 143, in deepcopy
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] y = copier(memo)
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] ^^^^^^^^^^^^
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] File “/home/jason/myai2/lib/python3.12/site-packages/torch/fx/graph_module.py”, line 889, in deepcopy
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] fake_mod = _CodeOnlyModule(copy.deepcopy(self.dict, memo))
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] File “/usr/lib/python3.12/copy.py”, line 136, in deepcopy
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] y = copier(x, memo)
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] ^^^^^^^^^^^^^^^
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] File “/usr/lib/python3.12/copy.py”, line 221, in _deepcopy_dict
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] y[deepcopy(key, memo)] = deepcopy(value, memo)
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] ^^^^^^^^^^^^^^^^^^^^^
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] File “/usr/lib/python3.12/copy.py”, line 136, in deepcopy
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] y = copier(x, memo)
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] ^^^^^^^^^^^^^^^
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] File “/usr/lib/python3.12/copy.py”, line 221, in _deepcopy_dict
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] y[deepcopy(key, memo)] = deepcopy(value, memo)
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] ^^^^^^^^^^^^^^^^^^^^^
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] File “/usr/lib/python3.12/copy.py”, line 143, in deepcopy
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] y = copier(memo)
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] ^^^^^^^^^^^^
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] File “/home/jason/myai2/lib/python3.12/site-packages/torch/_tensor.py”, line 172, in deepcopy
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] new_storage = self._typed_storage()._deepcopy(memo)
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] File “/home/jason/myai2/lib/python3.12/site-packages/torch/storage.py”, line 1135, in _deepcopy
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] return self._new_wrapped_storage(copy.deepcopy(self._untyped_storage, memo))
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] File “/usr/lib/python3.12/copy.py”, line 143, in deepcopy
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] y = copier(memo)
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] ^^^^^^^^^^^^
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] File “/home/jason/myai2/lib/python3.12/site-packages/torch/storage.py”, line 239, in deepcopy
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] new_storage = self.clone()
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] ^^^^^^^^^^^^
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] File “/home/jason/myai2/lib/python3.12/site-packages/torch/storage.py”, line 253, in clone
W0423 07:47:56.457000 2930 torch/_functorch/aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] return type(self)(self.nbytes(), device=self.device).copy(self)
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] File “/home/jason/myai2/lib/python3.12/site-packages/torch/utils/_stats.py”, line 27, in wrapper
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] return fn(*args, **kwargs)
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] ^^^^^^^^^^^^^^^^^^^
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] File “/home/jason/myai2/lib/python3.12/site-packages/torch/_subclasses/fake_tensor.py”, line 1312, in torch_dispatch
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] return self.dispatch(func, types, args, kwargs)
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] File “/home/jason/myai2/lib/python3.12/site-packages/torch/_subclasses/fake_tensor.py”, line 1950, in dispatch
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] return self._cached_dispatch_impl(func, types, args, kwargs)
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] File “/home/jason/myai2/lib/python3.12/site-packages/torch/_subclasses/fake_tensor.py”, line 1424, in _cached_dispatch_impl
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] output = self._dispatch_impl(func, types, args, kwargs)
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] File “/home/jason/myai2/lib/python3.12/site-packages/torch/_subclasses/fake_tensor.py”, line 2609, in _dispatch_impl
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] r = func(*args, **kwargs)
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] ^^^^^^^^^^^^^^^^^^^^^
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] File “/home/jason/myai2/lib/python3.12/site-packages/torch/_ops.py”, line 806, in call
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] return self._op(*args, **kwargs)
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] ^^^^^^^^^^^^^^^^^^^^^^^^^
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] RuntimeError: Attempted to set the storage of a tensor on device “meta” to a storage on different device “cuda:0”. This is no longer allowed; the devices must match.