We are trying to use pipelined model parallelism to speed up model training. This work requires ourselves to manage different versions of weights. This is ok in version 1.4 pytorch, but the following error will be reported after 1.5，
In our scenario, we confirm that the version of the weight is reasonable， so Is there any upper-level API to disable the detection of the weight’s version
raceback (most recent call last):
File "bert_main.py", line 294, in <module>
main()
File "bert_main.py", line 210, in main
model_args.model_name_or_path) else None
File "/workspace/pipedream-1.5BW/lib/runtimecontrol.py", line 521, in train
self.runtime_control.run_forward_backward()
File "/workspace/pipedream-1.5BW/lib/runtimecontrol.py", line 194, in run_forward_backward
data_provider=data_provider)
File "/workspace/pipedream-1.5BW/lib/callableunit.py", line 157, in backward
result = self._backward(**kwargs)
File "/workspace/pipedream-1.5BW/lib/callableunit.py", line 317, in _backward
result = self._backward_g0(**kwargs)
File "/workspace/pipedream-1.5BW/lib/callableunit.py", line 392, in _backward_g0
return self._comm_backward(outputs, kwargs)
File "/workspace/pipedream-1.5BW/lib/callableunit.py", line 174, in _comm_backward
torch.autograd.backward(forward_output, backward_gradient)
File "/opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py", line 146, in backward
allow_unreachable=True, accumulate_grad=True) # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [768]] is at version 9; expected version 4 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).