We are trying to use pipelined model parallelism to speed up model training. This work requires ourselves to manage different versions of weights. This is ok in version 1.4 pytorch, but the following error will be reported after 1.5，
In our scenario, we confirm that the version of the weight is reasonable， so Is there any upper-level API to disable the detection of the weight’s version
raceback (most recent call last): File "bert_main.py", line 294, in <module> main() File "bert_main.py", line 210, in main model_args.model_name_or_path) else None File "/workspace/pipedream-1.5BW/lib/runtimecontrol.py", line 521, in train self.runtime_control.run_forward_backward() File "/workspace/pipedream-1.5BW/lib/runtimecontrol.py", line 194, in run_forward_backward data_provider=data_provider) File "/workspace/pipedream-1.5BW/lib/callableunit.py", line 157, in backward result = self._backward(**kwargs) File "/workspace/pipedream-1.5BW/lib/callableunit.py", line 317, in _backward result = self._backward_g0(**kwargs) File "/workspace/pipedream-1.5BW/lib/callableunit.py", line 392, in _backward_g0 return self._comm_backward(outputs, kwargs) File "/workspace/pipedream-1.5BW/lib/callableunit.py", line 174, in _comm_backward torch.autograd.backward(forward_output, backward_gradient) File "/opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py", line 146, in backward allow_unreachable=True, accumulate_grad=True) # allow_unreachable flag RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor ] is at version 9; expected version 4 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).