Hi,
Here is the error message:
Traceback (most recent call last):
File "/home/shiqian/zhangzh/FAFRCNN/train.py", line 28, in <module>
train()
File "/home/shiqian/zhangzh/FAFRCNN/train.py", line 24, in train
transfer_train_fast('/home/shiqian/zhangzh/FAFRCNN/checkpoints/fs_40tr_27.5con_d1_0.75l-0.20a-1.00b-1.00/03090209_0.310')
File "/home/shiqian/zhangzh/FAFRCNN/transfer_train_fast.py", line 82, in transfer_train_fast
adv_meters, meters, ins_adv_optimizer, img_adv_optimizer, train_num=cfg.dis_train_num)
File "/home/shiqian/zhangzh/FAFRCNN/dis_train.py", line 61, in dis_train
faster_rcnn_mp.rpn_features(t_images, t_features, t_targets)
File "/home/shiqian/zhangzh/FAFRCNN/model/faster_rcnn/faster_rcnn_model_parallel.py", line 269, in rpn_features
proposals, proposal_losses, iou, anchor_len = self.rpn(images, features, targets)
File "/home/shiqian/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 541, in __call__
result = self.forward(*input, **kwargs)
File "/home/shiqian/zhangzh/FAFRCNN/model/faster_rcnn/faster_rcnn_model_parallel.py", line 344, in forward
objectness.to(cfg.gpu5), pred_bbox_deltas.to(cfg.gpu5), labels, regression_targets)
File "/home/shiqian/.local/lib/python3.5/site-packages/torchvision/models/detection/rpn.py", line 371, in compute_loss
reduction="sum",
File "/home/shiqian/.local/lib/python3.5/site-packages/torch/nn/functional.py", line 2179, in l1_loss
ret = torch._C._nn.l1_loss(expanded_input, expanded_target, _Reduction.get_enum(reduction))
RuntimeError: std::bad_alloc: temporary_buffer::allocate: get_temporary_buffer failed
I encountered this error many times. It always happens at RPN’s loss computation. Sometimes I can train my model for many epochs without this error, but sometimes it occurred at the second epoch. I am training my model on a server, which should have enough memory. This is the output of free -h in bash:
total used free shared buff/cache available
Mem: 376G 103G 79G 10G 193G 259G
Swap: 7.6G 5.4G 2.2G
I modified the FasterRCNN in pytorch to enable model parallel manually, other components are basically the same.