"runtime error: load of value 190, which is not a valid value for type 'bool'" during loss computation with MarginRankingLoss and HingeEmbeddingLoss

1

I got this “runtime error: load of value 190, which is not a valid value for type ‘bool’” during loss computation with MarginRankingLoss and HingeEmbeddingLoss. It was working yesterday actually, but started to get these errors today.

The full stack trace is here,

pytorch/pytorch/torch/csrc/utils/python_arg_parser.h:292:51: runtime error: load of value 190, which is not a valid value for type 'bool'
    #0 0x7fcaf91a69bd in torch::PythonArgs::toBool(int) pytorch/pytorch/torch/csrc/utils/python_arg_parser.h:292
    #1 0x7fcaf9288dfa in torch::autograd::THPVariable_margin_ranking_loss(_object*, _object*, _object*) pytorch/pytorch/torch/csrc/autograd/generated/python_torch_functions.cpp:2988
    #2 0x7fcb13b481f3 in _PyCFunction_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/methodobject.c:231
    #3 0x7fcb13ba14ec in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5011
    #4 0x7fcb13b98784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #5 0x7fcb13ba1bc4 in _PyEval_EvalCodeWithName /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4340
    #6 0x7fcb13ba1bc4 in fast_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5152
    #7 0x7fcb13ba15d2 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5032
    #8 0x7fcb13b98784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #9 0x7fcb13ba3049 in _PyFunction_FastCall /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5093
    #10 0x7fcb13ba3049 in _PyFunction_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5195
    #11 0x7fcb13b0e46d in _PyObject_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2310
    #12 0x7fcb13b0ee01 in _PyObject_Call_Prepend /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2373
    #13 0x7fcb13b0e81b in PyObject_Call /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2261
    #14 0x7fcb13b9a4c0 in do_call_core /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5280
    #15 0x7fcb13b9a4c0 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3578
    #16 0x7fcb13ba3315 in _PyEval_EvalCodeWithName /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4340
    #17 0x7fcb13ba3315 in _PyFunction_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5244
    #18 0x7fcb13b0e46d in _PyObject_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2310
    #19 0x7fcb13b0ee01 in _PyObject_Call_Prepend /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2373
    #20 0x7fcb13b0e81b in PyObject_Call /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2261
    #21 0x7fcb13b5c5e8 in slot_tp_call /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/typeobject.c:6194
    #22 0x7fcb13b0e81b in PyObject_Call /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2261
    #23 0x7fcb13b9a4c0 in do_call_core /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5280
    #24 0x7fcb13b9a4c0 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3578
    #25 0x7fcb13ba1bc4 in _PyEval_EvalCodeWithName /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4340
    #26 0x7fcb13ba1bc4 in fast_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5152
    #27 0x7fcb13ba15d2 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5032
    #28 0x7fcb13b98784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #29 0x7fcb13ba1bc4 in _PyEval_EvalCodeWithName /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4340
    #30 0x7fcb13ba1bc4 in fast_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5152
    #31 0x7fcb13ba15d2 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5032
    #32 0x7fcb13b98784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #33 0x7fcb13ba18f9 in _PyFunction_FastCall /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5093
    #34 0x7fcb13ba18f9 in fast_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5128
    #35 0x7fcb13ba15d2 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5032
    #36 0x7fcb13b98784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #37 0x7fcb13b97892 in _PyEval_EvalCodeWithName /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4340
    #38 0x7fcb13b97892 in PyEval_EvalCodeEx /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4361
    #39 0x7fcb13b9761b in PyEval_EvalCode /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:878
    #40 0x7fcb13b95a88 in builtin_exec_impl /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/bltinmodule.c:983
    #41 0x7fcb13b95a88 in builtin_exec /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/clinic/bltinmodule.c.h:283
    #42 0x7fcb13b48174 in _PyCFunction_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/methodobject.c:234
    #43 0x7fcb13ba14ec in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5011
    #44 0x7fcb13b98784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #45 0x7fcb13ba1bc4 in _PyEval_EvalCodeWithName /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4340
    #46 0x7fcb13ba1bc4 in fast_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5152
    #47 0x7fcb13ba15d2 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5032
    #48 0x7fcb13b98784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #49 0x7fcb13ba1bc4 in _PyEval_EvalCodeWithName /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4340
    #50 0x7fcb13ba1bc4 in fast_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5152
    #51 0x7fcb13ba15d2 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5032
    #52 0x7fcb13b98784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #53 0x7fcb13ba2026 in _PyEval_EvalCodeWithName /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4340
    #54 0x7fcb13ba2026 in fast_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5152
    #55 0x7fcb13ba15d2 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5032
    #56 0x7fcb13b98784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #57 0x7fcb13b97892 in _PyEval_EvalCodeWithName /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4340
    #58 0x7fcb13b97892 in PyEval_EvalCodeEx /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4361
    #59 0x7fcb13b9761b in PyEval_EvalCode /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:878
    #60 0x7fcb13c23198 in run_mod /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/pythonrun.c:980
    #61 0x7fcb13c2313c in PyRun_StringFlags /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/pythonrun.c:904
    #62 0x7fcb13c23091 in PyRun_SimpleStringFlags /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/pythonrun.c:421
    #63 0x7fcb13c2a06c in run_command /home/engshare/third-party2/python/3.6/src/cpython-3.6/Modules/main.c:299
    #64 0x7fcb13c2a06c in Py_Main /home/engshare/third-party2/python/3.6/src/cpython-3.6/Modules/main.c:747
    #65 0x400afa in main (/usr/local/fbcode/gcc-5-glibc-2.23/bin/python3.6+0x400afa)
    #66 0x7fcb12d3c857 in __libc_start_main /home/engshare/third-party2/glibc/2.23/src/glibc-2.23/csu/../csu/libc-start.c:289
    #67 0x400c38 in _start (/usr/local/fbcode/gcc-5-glibc-2.23/bin/python3.6+0x400c38)

Does anyone happen to know why?

2

What were your inputs to margin_ranking_loss?

3

Since I was doing some kind of pairwise training. The input to margin_ranking_loss is (left_input, right_input, target). The left/right input are double tensors of size (batch_size, )

4

I’m not really sure what the error is. If you could provide sample inputs to MarginRankingLoss that trigger that error for you it’ll be easier to debug.

5

Here’s the sample inputs.

1.00000e-02 *
  6.5441
  1.6199
  1.6199
  1.6199
  0.9627
  0.9627
  0.9351
  1.6199
  1.6199
  1.6199
  1.6199
  2.8784
  0.9627
  0.9627
  4.0087
  0.9351
[torch.DoubleTensor of size (16,)]


1.00000e-02 *
  0.9627
  1.6199
  4.0087
  1.6199
  4.7620
  4.7620
  2.3129
  1.6199
  0.1195
  1.6199
  1.6199
  1.4622
  4.7620
  8.2307
  1.6199
  2.3129
[torch.DoubleTensor of size (16,)]


-1
-1
 1
-1
 1
 1
 1
 1
 1
-1
-1
-1
 1
 1
-1
 1
[torch.DoubleTensor of size (16,)]

The code does not fail on some specific inputs. It fails on all of them, so I’m not sure whether it is the input that is causing the problem.

6

Hi, https://github.com/pytorch/pytorch/pull/6254 should fix your issue.

Alternatively, you can add defaults to the boolean args here to circumvent this issue: https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/native_functions.yaml#L425
(also for HingeEmbedding)

1 Like
7

I’m also running into this error, but during a forward pass. I pulled in the proposed fix, but it doesn’t fix my issue.

The input to the forward pass is:

(Pdb) x

-0.4636 -0.8860  0.5684
[torch.FloatTensor of size (1,3)]

The crash happens when this tensor gets passed into a batch norm layer. Note, my model is in eval() mode for this. Here is my model and the batch norm layer that causes the crash:

# Model

(Pdb) self
ActorNet(
  (layers): ModuleList(
    (0): Linear(in_features=3, out_features=400, bias=True)
    (1): Linear(in_features=400, out_features=300, bias=True)
    (2): Linear(in_features=300, out_features=1, bias=True)
  )
  (batch_norm_ops): ModuleList(
    (0): BatchNorm1d(3, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (1): BatchNorm1d(400, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): BatchNorm1d(300, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  )
)

# Batch norm layer
(Pdb) self.batch_norm_ops[i]
BatchNorm1d(3, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)

Code that causes the crash:

x = self.batch_norm_ops[i](x)

Crash dump:

pytorch/pytorch/torch/csrc/utils/python_arg_parser.h:294:51: runtime error: load of value 190, which is not a valid value for type 'bool'
    #0 0x7f274726565d in torch::PythonArgs::toBool(int) pytorch/pytorch/torch/csrc/utils/python_arg_parser.h:294
    #1 0x7f27472ec760 in torch::autograd::THPVariable_batch_norm(_object*, _object*, _object*) pytorch/pytorch/torch/csrc/autograd/generated/python_torch_functions.cpp:1426
    #2 0x7f27a18a01f3 in _PyCFunction_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/methodobject.c:231
    #3 0x7f27a18f94ec in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5011
    #4 0x7f27a18f0784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #5 0x7f27a18f9bc4 in _PyEval_EvalCodeWithName /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4340
    #6 0x7f27a18f9bc4 in fast_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5152
    #7 0x7f27a18f95d2 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5032
    #8 0x7f27a18f0784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #9 0x7f27a18fb049 in _PyFunction_FastCall /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5093
    #10 0x7f27a18fb049 in _PyFunction_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5195
    #11 0x7f27a186646d in _PyObject_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2310
    #12 0x7f27a1866e01 in _PyObject_Call_Prepend /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2373
    #13 0x7f27a186681b in PyObject_Call /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2261
    #14 0x7f27a18f24c0 in do_call_core /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5280
    #15 0x7f27a18f24c0 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3578
    #16 0x7f27a18fb315 in _PyEval_EvalCodeWithName /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4340
    #17 0x7f27a18fb315 in _PyFunction_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5244
    #18 0x7f27a186646d in _PyObject_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2310
    #19 0x7f27a1866e01 in _PyObject_Call_Prepend /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2373
    #20 0x7f27a186681b in PyObject_Call /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2261
    #21 0x7f27a18b45e8 in slot_tp_call /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/typeobject.c:6194
    #22 0x7f27a18663ec in _PyObject_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2331
    #23 0x7f27a18f9679 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5035
    #24 0x7f27a18f0784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #25 0x7f27a18ef892 in _PyEval_EvalCodeWithName /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4340
    #26 0x7f27a18ef892 in PyEval_EvalCodeEx /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4361
    #27 0x7f27a18ef61b in PyEval_EvalCode /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:878
    #28 0x7f27a18eda88 in builtin_exec_impl /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/bltinmodule.c:983
    #29 0x7f27a18eda88 in builtin_exec /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/clinic/bltinmodule.c.h:283
    #30 0x7f27a18a0174 in _PyCFunction_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/methodobject.c:234
    #31 0x7f27a18f94ec in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5011
    #32 0x7f27a18f0784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #33 0x7f27a18f98f9 in _PyFunction_FastCall /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5093
    #34 0x7f27a18f98f9 in fast_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5128
    #35 0x7f27a18f95d2 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5032
    #36 0x7f27a18f0784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #37 0x7f27a18f98f9 in _PyFunction_FastCall /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5093
    #38 0x7f27a18f98f9 in fast_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5128
    #39 0x7f27a18f95d2 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5032
    #40 0x7f27a18f0784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #41 0x7f27a18f98f9 in _PyFunction_FastCall /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5093
    #42 0x7f27a18f98f9 in fast_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5128
    #43 0x7f27a18f95d2 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5032
    #44 0x7f27a18f0784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #45 0x7f27a18f9bc4 in _PyEval_EvalCodeWithName /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4340
    #46 0x7f27a18f9bc4 in fast_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5152
    #47 0x7f27a18f95d2 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5032
    #48 0x7f27a18f0784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #49 0x7f27a18f98f9 in _PyFunction_FastCall /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5093
    #50 0x7f27a18f98f9 in fast_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5128
    #51 0x7f27a18f95d2 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5032
    #52 0x7f27a18f0784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #53 0x7f27a18f98f9 in _PyFunction_FastCall /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5093
    #54 0x7f27a18f98f9 in fast_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5128
    #55 0x7f27a18f95d2 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5032
    #56 0x7f27a18f0784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #57 0x7f27a18f98f9 in _PyFunction_FastCall /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5093
    #58 0x7f27a18f98f9 in fast_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5128
    #59 0x7f27a18f95d2 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5032
    #60 0x7f27a18f0784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #61 0x7f27a18f98f9 in _PyFunction_FastCall /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5093
    #62 0x7f27a18f98f9 in fast_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5128
    #63 0x7f27a18f95d2 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5032
    #64 0x7f27a18f0784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #65 0x7f27a18fb049 in _PyFunction_FastCall /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5093
    #66 0x7f27a18fb049 in _PyFunction_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5195
    #67 0x7f27a186646d in _PyObject_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2310
    #68 0x7f27a1866e01 in _PyObject_Call_Prepend /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2373
    #69 0x7f27a18663ec in _PyObject_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2331
    #70 0x7f27a191fdbc in call_trampoline /home/engshare/third-party2/python/3.6/src/cpython-3.6/./Python/sysmodule.c:400
    #71 0x7f27a191fdbc in trace_trampoline /home/engshare/third-party2/python/3.6/src/cpython-3.6/./Python/sysmodule.c:440
    #72 0x7f27a18f6780 in call_trace /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4696
    #73 0x7f27a18f6780 in maybe_call_line_trace /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4744
    #74 0x7f27a18f6780 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:1385
    #75 0x7f27a18fb049 in _PyFunction_FastCall /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5093
    #76 0x7f27a18fb049 in _PyFunction_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5195
    #77 0x7f27a186646d in _PyObject_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2310
    #78 0x7f27a1866e01 in _PyObject_Call_Prepend /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2373
    #79 0x7f27a186681b in PyObject_Call /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2261
    #80 0x7f27a18f24c0 in do_call_core /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5280
    #81 0x7f27a18f24c0 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3578
    #82 0x7f27a18fb315 in _PyEval_EvalCodeWithName /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4340
    #83 0x7f27a18fb315 in _PyFunction_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5244
    #84 0x7f27a186646d in _PyObject_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2310
    #85 0x7f27a1866e01 in _PyObject_Call_Prepend /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2373
    #86 0x7f27a186681b in PyObject_Call /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2261
    #87 0x7f27a18b45e8 in slot_tp_call /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/typeobject.c:6194
    #88 0x7f27a18663ec in _PyObject_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2331
    #89 0x7f27a18f9679 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5035
    #90 0x7f27a18f0784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #91 0x7f27a18fa026 in _PyEval_EvalCodeWithName /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4340
    #92 0x7f27a18fa026 in fast_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5152
    #93 0x7f27a18f95d2 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5032
    #94 0x7f27a18f1947 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3525
    #95 0x7f27a18f98f9 in _PyFunction_FastCall /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5093
    #96 0x7f27a18f98f9 in fast_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5128
    #97 0x7f27a18f95d2 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5032
    #98 0x7f27a18f0784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #99 0x7f27a18fa026 in _PyEval_EvalCodeWithName /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4340
    #100 0x7f27a18fa026 in fast_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5152
    #101 0x7f27a18f95d2 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5032
    #102 0x7f27a18f0784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #103 0x7f27a18ef892 in _PyEval_EvalCodeWithName /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4340
    #104 0x7f27a18ef892 in PyEval_EvalCodeEx /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4361
    #105 0x7f27a187bb60 in function_call /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/funcobject.c:604
    #106 0x7f27a186681b in PyObject_Call /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2261
    #107 0x7f27a18f24c0 in do_call_core /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5280
    #108 0x7f27a18f24c0 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3578
    #109 0x7f27a18f98f9 in _PyFunction_FastCall /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5093
    #110 0x7f27a18f98f9 in fast_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5128
    #111 0x7f27a18f95d2 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5032
    #112 0x7f27a18f0784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #113 0x7f27a18f98f9 in _PyFunction_FastCall /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5093
    #114 0x7f27a18f98f9 in fast_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5128
    #115 0x7f27a18f95d2 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5032
    #116 0x7f27a18f0784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #117 0x7f27a18f98f9 in _PyFunction_FastCall /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5093
    #118 0x7f27a18f98f9 in fast_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5128
    #119 0x7f27a18f95d2 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5032
    #120 0x7f27a18f0784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #121 0x7f27a18fb315 in _PyEval_EvalCodeWithName /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4340
    #122 0x7f27a18fb315 in _PyFunction_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5244
    #123 0x7f27a186646d in _PyObject_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2310
    #124 0x7f27a1866e01 in _PyObject_Call_Prepend /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2373
    #125 0x7f27a186681b in PyObject_Call /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2261
    #126 0x7f27a18f24c0 in do_call_core /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5280
    #127 0x7f27a18f24c0 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3578
    #128 0x7f27a18fb315 in _PyEval_EvalCodeWithName /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4340
    #129 0x7f27a18fb315 in _PyFunction_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5244
    #130 0x7f27a186646d in _PyObject_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2310
    #131 0x7f27a1866e01 in _PyObject_Call_Prepend /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2373
    #132 0x7f27a186681b in PyObject_Call /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2261
    #133 0x7f27a18b45e8 in slot_tp_call /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/typeobject.c:6194
    #134 0x7f27a18663ec in _PyObject_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2331
    #135 0x7f27a18f9679 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5035
    #136 0x7f27a18f0784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #137 0x7f27a18fb315 in _PyEval_EvalCodeWithName /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4340
    #138 0x7f27a18fb315 in _PyFunction_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5244
    #139 0x7f27a186646d in _PyObject_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2310
    #140 0x7f27a1866e01 in _PyObject_Call_Prepend /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2373
    #141 0x7f27a186681b in PyObject_Call /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2261
    #142 0x7f27a18f24c0 in do_call_core /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5280
    #143 0x7f27a18f24c0 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3578
    #144 0x7f27a18fb315 in _PyEval_EvalCodeWithName /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4340
    #145 0x7f27a18fb315 in _PyFunction_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5244
    #146 0x7f27a186646d in _PyObject_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2310
    #147 0x7f27a1866e01 in _PyObject_Call_Prepend /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2373
    #148 0x7f27a186681b in PyObject_Call /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2261
    #149 0x7f27a18b45e8 in slot_tp_call /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/typeobject.c:6194
    #150 0x7f27a18663ec in _PyObject_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2331
    #151 0x7f27a18f9679 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5035
    #152 0x7f27a18f0784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #153 0x7f27a18fb315 in _PyEval_EvalCodeWithName /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4340
    #154 0x7f27a18fb315 in _PyFunction_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5244
    #155 0x7f27a186646d in _PyObject_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2310
    #156 0x7f27a1866e01 in _PyObject_Call_Prepend /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2373
    #157 0x7f27a186681b in PyObject_Call /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2261
    #158 0x7f27a18f24c0 in do_call_core /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5280
    #159 0x7f27a18f24c0 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3578
    #160 0x7f27a18fb315 in _PyEval_EvalCodeWithName /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4340
    #161 0x7f27a18fb315 in _PyFunction_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5244
    #162 0x7f27a186646d in _PyObject_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2310
    #163 0x7f27a1866e01 in _PyObject_Call_Prepend /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2373
    #164 0x7f27a186681b in PyObject_Call /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2261
    #165 0x7f27a18b45e8 in slot_tp_call /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/typeobject.c:6194
    #166 0x7f27a18663ec in _PyObject_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/abstract.c:2331
    #167 0x7f27a18f9679 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5035
    #168 0x7f27a18f0784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #169 0x7f27a18f98f9 in _PyFunction_FastCall /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5093
    #170 0x7f27a18f98f9 in fast_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5128
    #171 0x7f27a18f95d2 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5032
    #172 0x7f27a18f0784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #173 0x7f27a18f98f9 in _PyFunction_FastCall /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5093
    #174 0x7f27a18f98f9 in fast_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5128
    #175 0x7f27a18f95d2 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5032
    #176 0x7f27a18f0784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #177 0x7f27a18f98f9 in _PyFunction_FastCall /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5093
    #178 0x7f27a18f98f9 in fast_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5128
    #179 0x7f27a18f95d2 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5032
    #180 0x7f27a18f0784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #181 0x7f27a18f98f9 in _PyFunction_FastCall /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5093
    #182 0x7f27a18f98f9 in fast_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5128
    #183 0x7f27a18f95d2 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5032
    #184 0x7f27a18f0784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #185 0x7f27a18ef892 in _PyEval_EvalCodeWithName /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4340
    #186 0x7f27a18ef892 in PyEval_EvalCodeEx /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4361
    #187 0x7f27a18ef61b in PyEval_EvalCode /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:878
    #188 0x7f27a18eda88 in builtin_exec_impl /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/bltinmodule.c:983
    #189 0x7f27a18eda88 in builtin_exec /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/clinic/bltinmodule.c.h:283
    #190 0x7f27a18a0174 in _PyCFunction_FastCallDict /home/engshare/third-party2/python/3.6/src/cpython-3.6/Objects/methodobject.c:234
    #191 0x7f27a18f94ec in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5011
    #192 0x7f27a18f0784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #193 0x7f27a18f9bc4 in _PyEval_EvalCodeWithName /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4340
    #194 0x7f27a18f9bc4 in fast_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5152
    #195 0x7f27a18f95d2 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5032
    #196 0x7f27a18f0784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #197 0x7f27a18f9bc4 in _PyEval_EvalCodeWithName /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4340
    #198 0x7f27a18f9bc4 in fast_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5152
    #199 0x7f27a18f95d2 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5032
    #200 0x7f27a18f0784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #201 0x7f27a18fa026 in _PyEval_EvalCodeWithName /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4340
    #202 0x7f27a18fa026 in fast_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5152
    #203 0x7f27a18f95d2 in call_function /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:5032
    #204 0x7f27a18f0784 in _PyEval_EvalFrameDefault /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:3509
    #205 0x7f27a18ef892 in _PyEval_EvalCodeWithName /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4340
    #206 0x7f27a18ef892 in PyEval_EvalCodeEx /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:4361
    #207 0x7f27a18ef61b in PyEval_EvalCode /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/ceval.c:878
    #208 0x7f27a197b198 in run_mod /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/pythonrun.c:980
    #209 0x7f27a197b13c in PyRun_StringFlags /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/pythonrun.c:904
    #210 0x7f27a197b091 in PyRun_SimpleStringFlags /home/engshare/third-party2/python/3.6/src/cpython-3.6/Python/pythonrun.c:421
    #211 0x7f27a198206c in run_command /home/engshare/third-party2/python/3.6/src/cpython-3.6/Modules/main.c:299
    #212 0x7f27a198206c in Py_Main /home/engshare/third-party2/python/3.6/src/cpython-3.6/Modules/main.c:747
    #213 0x400afa in main (/usr/local/fbcode/gcc-5-glibc-2.23/bin/python3.6+0x400afa)
    #214 0x7f27a0a94857 in __libc_start_main /home/engshare/third-party2/glibc/2.23/src/glibc-2.23/csu/../csu/libc-start.c:289
    #215 0x400c38 in _start (/usr/local/fbcode/gcc-5-glibc-2.23/bin/python3.6+0x400c38)

SUMMARY: UndefinedBehaviorSanitizer: invalid-bool-load pytorch/pytorch/torch/csrc/utils/python_arg_parser.h:294:51 in
8

By any chance did you not get the PR? Your trace looks like it’s from the older version. (python_arg_parse.h:294 is a blank line in the new one)

9

That’s possible. Taking a look now.