RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [100, 2, 6]], which is output 0 of SigmoidBackward0, is at version 6; expected version 0 instead. Hint: the backtrace fur

talonx1 · October 10, 2023, 8:23am

The error info:

Epoch 0:   0%|                                         | 0/1069 [00:00<?, ?it/s]/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/torch/autograd/__init__.py:173: UserWarning: Error detected in SigmoidBackward0. Traceback of forward call that caused the error:
  File "/data/qinhaobo/Mask3D-main/det_debug_main_instance_segmentation.py", line 134, in <module>
    main()
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/hydra/main.py", line 32, in decorated_main
    _run_hydra(
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/hydra/_internal/utils.py", line 346, in _run_hydra
    run_and_report(
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/hydra/_internal/utils.py", line 198, in run_and_report
    return func()
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/hydra/_internal/utils.py", line 347, in <lambda>
    lambda: hydra.run(
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 107, in run
    return run_job(
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/hydra/core/utils.py", line 128, in run_job
    ret.return_value = task_function(task_cfg)
  File "/data/qinhaobo/Mask3D-main/det_debug_main_instance_segmentation.py", line 128, in main
    train(cfg)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/hydra/main.py", line 27, in decorated_main
    return task_function(cfg_passthrough)
  File "/data/qinhaobo/Mask3D-main/det_debug_main_instance_segmentation.py", line 109, in train
    runner.fit(model)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 697, in fit
    self._call_and_handle_interrupt(
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 651, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 738, in _fit_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1169, in _run
    results = self._run_stage()
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1255, in _run_stage
    return self._run_train()
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1286, in _run_train
    self.fit_loop.run()
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
    self.advance(*args, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 270, in advance
    self._outputs = self.epoch_loop.run(self._data_fetcher)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
    self.advance(*args, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 203, in advance
    batch_output = self.batch_loop.run(kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
    self.advance(*args, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 87, in advance
    outputs = self.optimizer_loop.run(optimizers, kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
    self.advance(*args, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 201, in advance
    result = self._run_optimization(kwargs, self._optimizers[self.optim_progress.optimizer_position])
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 248, in _run_optimization
    self._optimizer_step(optimizer, opt_idx, kwargs.get("batch_idx", 0), closure)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 358, in _optimizer_step
    self.trainer._call_lightning_module_hook(
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1553, in _call_lightning_module_hook
    output = fn(*args, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/core/module.py", line 1673, in optimizer_step
    optimizer.step(closure=optimizer_closure)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/core/optimizer.py", line 168, in step
    step_output = self._strategy.optimizer_step(self._optimizer, self._optimizer_idx, closure, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/strategies/strategy.py", line 216, in optimizer_step
    return self.precision_plugin.optimizer_step(model, optimizer, opt_idx, closure, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 153, in optimizer_step
    return optimizer.step(closure=closure, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/torch/optim/lr_scheduler.py", line 65, in wrapper
    return wrapped(*args, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/torch/optim/optimizer.py", line 113, in wrapper
    return func(*args, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/torch/optim/adamw.py", line 119, in step
    loss = closure()
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 138, in _wrap_closure
    closure_result = closure()
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 146, in __call__
    self._result = self.closure(*args, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 132, in closure
    step_output = self._step_fn()
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 407, in _training_step
    training_step_output = self.trainer._call_strategy_hook("training_step", *kwargs.values())
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1707, in _call_strategy_hook
    output = fn(*args, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/strategies/strategy.py", line 358, in training_step
    return self.model.training_step(*args, **kwargs)
  File "/data/qinhaobo/Mask3D-main/trainer/trainer_det.py", line 130, in training_step
    output = self.forward(data,
  File "/data/qinhaobo/Mask3D-main/trainer/trainer_det.py", line 105, in forward
    x = self.model(x, point2segment, raw_coordinates=raw_coordinates,     
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/data/qinhaobo/Mask3D-main/models/detmask3d.py", line 667, in forward
    decoder_out, mask_dict = self.det_transformer_decoder(
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/data/qinhaobo/Mask3D-main/models/modules/maskdino_decoder.py", line 622, in forward
    hs, references = self.decoder(
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/data/qinhaobo/Mask3D-main/models/modules/detr/dab_transformer.py", line 251, in forward
    new_reference_points = outputs_unsig.sigmoid() 
 (Triggered internally at  ../torch/csrc/autograd/python_anomaly_mode.cpp:102.)
  Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
Traceback (most recent call last):
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/hydra/_internal/utils.py", line 198, in run_and_report
    return func()
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/hydra/_internal/utils.py", line 347, in <lambda>
    lambda: hydra.run(
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 107, in run
    return run_job(
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/hydra/core/utils.py", line 128, in run_job
    ret.return_value = task_function(task_cfg)
  File "/data/qinhaobo/Mask3D-main/det_debug_main_instance_segmentation.py", line 128, in main
    train(cfg)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/hydra/main.py", line 27, in decorated_main
    return task_function(cfg_passthrough)
  File "/data/qinhaobo/Mask3D-main/det_debug_main_instance_segmentation.py", line 109, in train
    runner.fit(model)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 697, in fit
    self._call_and_handle_interrupt(
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 651, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 738, in _fit_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1169, in _run
    results = self._run_stage()
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1255, in _run_stage
    return self._run_train()
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1286, in _run_train
    self.fit_loop.run()
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
    self.advance(*args, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 270, in advance
    self._outputs = self.epoch_loop.run(self._data_fetcher)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
    self.advance(*args, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 203, in advance
    batch_output = self.batch_loop.run(kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
    self.advance(*args, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 87, in advance
    outputs = self.optimizer_loop.run(optimizers, kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
    self.advance(*args, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 201, in advance
    result = self._run_optimization(kwargs, self._optimizers[self.optim_progress.optimizer_position])
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 248, in _run_optimization
    self._optimizer_step(optimizer, opt_idx, kwargs.get("batch_idx", 0), closure)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 358, in _optimizer_step
    self.trainer._call_lightning_module_hook(
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1553, in _call_lightning_module_hook
    output = fn(*args, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/core/module.py", line 1673, in optimizer_step
    optimizer.step(closure=optimizer_closure)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/core/optimizer.py", line 168, in step
    step_output = self._strategy.optimizer_step(self._optimizer, self._optimizer_idx, closure, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/strategies/strategy.py", line 216, in optimizer_step
    return self.precision_plugin.optimizer_step(model, optimizer, opt_idx, closure, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 153, in optimizer_step
    return optimizer.step(closure=closure, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/torch/optim/lr_scheduler.py", line 65, in wrapper
    return wrapped(*args, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/torch/optim/optimizer.py", line 113, in wrapper
    return func(*args, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/torch/optim/adamw.py", line 119, in step
    loss = closure()
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 138, in _wrap_closure
    closure_result = closure()
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 146, in __call__
    self._result = self.closure(*args, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 141, in closure
    self._backward_fn(step_output.closure_loss)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 304, in backward_fn
    self.trainer._call_strategy_hook("backward", loss, optimizer, opt_idx)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1707, in _call_strategy_hook
    output = fn(*args, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/strategies/strategy.py", line 191, in backward
    self.precision_plugin.backward(self.lightning_module, closure_loss, optimizer, optimizer_idx, *args, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 80, in backward
    model.backward(closure_loss, optimizer, optimizer_idx, *args, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/core/module.py", line 1418, in backward
    loss.backward(*args, **kwargs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/torch/_tensor.py", line 396, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/data/qinhaobo/anaconda3/envs/mask3d/lib/python3.10/site-packages/torch/autograd/__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [100, 2, 6]], which is output 0 of SigmoidBackward0, is at version 6; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

During handling of the above exception, another exception occurred:

related code:

 if self.bbox_embed is not None:
                reference_before_sigmoid = inverse_sigmoid(reference_boxes) 
                offsets_unsig = self.bbox_embed[idx](query)  
                outputs_unsig = offsets_unsig + reference_before_sigmoid 
                new_reference_points = outputs_unsig.sigmoid()
                reference_boxes = new_reference_points.detach()
                intermediate_ref_boxes.append(new_reference_points)

KFrank · October 11, 2023, 8:54pm

Hi Talon!

talonx1:

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [100, 2, 6]], which is output 0 of SigmoidBackward0, is at version 6; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

The error message gives you information that can point you in the right direction.

The tensor that is being modified inplace has shape [100, 2, 6] and appears
to be being passed into a Sigmoid. See if you can find that tensor in your code.

The error message complains about “version 6,” so the tensor is being modified
inplace six times before you call .backward(). That may give you a hint about
what operation is performing the inplace modification. You can also print out
the ._version property of the suspect tensor at various places in your code
and use a divide-and-conquer scheme to track down where exactly the tensor
is being modified.

Here’s a post that discusses some debugging techniques for these kinds of
errors:

"RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [64, 1]], which is output 0 of AsStridedBackward0, is at version 3; expected version 2 instead. Hint: the backtrace further a autograd

Hi Fahmyadan and Sangyoon! Here are some suggestions about how to track down (and maybe fix) inplace-modification errors. Note that an inplace modification in the forward pass is not necessarily* an error – it depends on whether and how the tensor that was modified is used in the backward pass. Note that inplace operations can be useful for saving memory – if you replace an innocent inplace operation with an out-of-place equivalent, your training will use more memory (and, to a minor e…

Good luck!

K. Frank