I am getting below message and numa test failed, Can someone please guide what going worng here?
numactl --hardware
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6 7 32 33 34 35 36 37 38 39
node 0 size: 64351 MB
node 0 free: 1532 MB
node 1 cpus: 8 9 10 11 12 13 14 15 40 41 42 43 44 45 46 47
node 1 size: 64505 MB
node 1 free: 551 MB
node 2 cpus: 16 17 18 19 20 21 22 23 48 49 50 51 52 53 54 55
node 2 size: 64480 MB
node 2 free: 17961 MB
node 3 cpus: 24 25 26 27 28 29 30 31 56 57 58 59 60 61 62 63
node 3 size: 64504 MB
node 3 free: 32454 MB
node distances:
node 0 1 2 3
0: 10 16 16 16
1: 16 10 16 16
2: 16 16 10 16
3: 16 16 16 10
Failure log
…/…/…/…/root/.local/lib/python2.7/site-packages/caffe2/python/numa_test.py::NUMATest::test_numa FAILED [ 15%]
=================================== FAILURES ===================================
______________________________ NUMATest.test_numa ______________________________
self = <caffe2.python.numa_test.NUMATest testMethod=test_numa>
def test_numa(self):
net = build_test_net("test_numa")
workspace.RunNetOnce(net)
/root/.local/lib/python2.7/site-packages/caffe2/python/numa_test.py:46:
/root/.local/lib/python2.7/site-packages/caffe2/python/workspace.py:232: in RunNetOnce
StringifyProto(net),
func = <built-in method run_net_once of PyCapsule object at 0x7f685e201090>
op_id_fetcher = <bound method Workspace._last_failed_op_net_position of <caffe2.python.caffe2_pybind11_state_hip.Workspace object at 0x7f635c40a378>>
net_name = ‘test_numa’
args = (‘\n\ttest_numa\x12>\x12\routput_blob_0\x1a\x00"\x0cConstantFill*\t\n\x05shape0\x01*\x0c\n\x05value\x15\xc3\xf5H@2\x04…x128\n\routput_blob_1\x12\x11output_blob_1_gpu\x1a\x00"\x0cCopyCPUToGPU2\x04\x08\x01\x10\x00\x1a\x10async_scheduling’,)
kwargs = {}, op_id = 2
net_tracebacks = {0: [(‘/root/.local/lib/python2.7/site-packages/caffe2/python/numa_test.py’, 21, ‘build_test_net’), (‘/root/.local/lib…t.py’, 221, ‘runtest’), (‘/usr/local/lib/python2.7/dist-packages/_pytest/runner.py’, 122, ‘pytest_runtest_call’), …]}
tb = [(‘/root/.local/lib/python2.7/site-packages/caffe2/python/numa_test.py’, 32, ‘build_test_net’), (‘/root/.local/lib/pyt…st.py’, 221, ‘runtest’), (‘/usr/local/lib/python2.7/dist-packages/_pytest/runner.py’, 122, ‘pytest_runtest_call’), …]
line = (‘/root/.local/lib/python2.7/site-packages/caffe2/python/numa_test.py’, 32, ‘build_test_net’)
def CallWithExceptionIntercept(func, op_id_fetcher, net_name, *args, **kwargs):
try:
return func(*args, **kwargs)
E RuntimeError: [enforce fail at operator.cc:243] op. Cannot create operator of type ‘CopyCPUToGPU’ on the device ‘CUDA’. Verify that implementation for the corresponding device exist. It might also happen if the binary is not linked with the operator implementation code. If Python frontend is used it might happen if dyndep.InitOpsLibrary call is missing. Operator def: input: “output_blob_0” output: “output_blob_0_gpu” name: “” type: “CopyCPUToGPU” device_option { device_type: 1 device_id: 0 }
E frame #0: c10::ThrowEnforceNotMet(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, void const*) + 0xa3 (0x7f6816937783 in /root/.local/lib/python2.7/site-packages/caffe2/python/…/…/torch/lib/libc10.so)
E frame #1: caffe2::CreateOperator(caffe2::OperatorDef const&, caffe2::Workspace*, int) + 0xd41 (0x7f6818916161 in /root/.local/lib/python2.7/site-packages/caffe2/python/…/…/torch/lib/libcaffe2.so)
E frame #2: caffe2::dag_utils::prepareOperatorNodes(std::shared_ptr<caffe2::NetDef const> const&, caffe2::Workspace*) + 0x321 (0x7f6818903a91 in /root/.local/lib/python2.7/site-packages/caffe2/python/…/…/torch/lib/libcaffe2.so)
E frame #3: caffe2::AsyncNetBase::AsyncNetBase(std::shared_ptr<caffe2::NetDef const> const&, caffe2::Workspace*) + 0x1ad (0x7f68188eb4dd in /root/.local/lib/python2.7/site-packages/caffe2/python/…/…/torch/lib/libcaffe2.so)
E frame #4: + 0x1b9507c (0x7f68188f507c in /root/.local/lib/python2.7/site-packages/caffe2/python/…/…/torch/lib/libcaffe2.so)
E frame #5: + 0x1b951e2 (0x7f68188f51e2 in /root/.local/lib/python2.7/site-packages/caffe2/python/…/…/torch/lib/libcaffe2.so)
E frame #6: caffe2::CreateNet(std::shared_ptr<caffe2::NetDef const> const&, caffe2::Workspace*) + 0x3a2 (0x7f68188e7552 in /root/.local/lib/python2.7/site-packages/caffe2/python/…/…/torch/lib/libcaffe2.so)
E frame #7: caffe2::CreateNet(caffe2::NetDef const&, caffe2::Workspace*) + 0x92 (0x7f68188e6f92 in /root/.local/lib/python2.7/site-packages/caffe2/python/…/…/torch/lib/libcaffe2.so)
E frame #8: caffe2::Workspace::RunNetOnce(caffe2::NetDef const&) + 0x21 (0x7f681893f681 in /root/.local/lib/python2.7/site-packages/caffe2/python/…/…/torch/lib/libcaffe2.so)
E frame #9: + 0x4705d (0x7f685dee405d in /root/.local/lib/python2.7/site-packages/caffe2/python/caffe2_pybind11_state_hip.so)
E frame #10: + 0x6c0bb (0x7f685df090bb in /root/.local/lib/python2.7/site-packages/caffe2/python/caffe2_pybind11_state_hip.so)
E frame #11: PyEval_EvalFrameEx + 0x6f3e (0x4c2e1e in /usr/bin/python2)
E frame #12: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
E frame #13: PyEval_EvalFrameEx + 0x6076 (0x4c1f56 in /usr/bin/python2)
E frame #14: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
E frame #15: PyEval_EvalFrameEx + 0x6076 (0x4c1f56 in /usr/bin/python2)
E frame #16: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
E frame #17: PyEval_EvalFrameEx + 0x6076 (0x4c1f56 in /usr/bin/python2)
E frame #18: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
E frame #19: /usr/bin/python2() [0x4d57a3]
E frame #20: PyObject_Call + 0x3e (0x4a587e in /usr/bin/python2)
E frame #21: PyEval_EvalFrameEx + 0x263e (0x4be51e in /usr/bin/python2)
E frame #22: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
E frame #23: /usr/bin/python2() [0x4d57a3]
E frame #24: /usr/bin/python2() [0x4eef5e]
E frame #25: PyObject_Call + 0x3e (0x4a587e in /usr/bin/python2)
E frame #26: /usr/bin/python2() [0x548fc3]
E frame #27: PyEval_EvalFrameEx + 0x578d (0x4c166d in /usr/bin/python2)
E frame #28: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
E frame #29: PyEval_EvalFrameEx + 0x6076 (0x4c1f56 in /usr/bin/python2)
E frame #30: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
E frame #31: /usr/bin/python2() [0x4d5669]
E frame #32: PyObject_Call + 0x3e (0x4a587e in /usr/bin/python2)
E frame #33: PyEval_EvalFrameEx + 0x263e (0x4be51e in /usr/bin/python2)
E frame #34: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
E frame #35: PyEval_EvalFrameEx + 0x58e6 (0x4c17c6 in /usr/bin/python2)
E frame #36: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
E frame #37: PyEval_EvalFrameEx + 0x6076 (0x4c1f56 in /usr/bin/python2)
E frame #38: PyEval_EvalFrameEx + 0x553f (0x4c141f in /usr/bin/python2)
E frame #39: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
E frame #40: /usr/bin/python2() [0x4d57a3]
E frame #41: /usr/bin/python2() [0x4eef5e]
E frame #42: PyObject_Call + 0x3e (0x4a587e in /usr/bin/python2)
E frame #43: /usr/bin/python2() [0x548fc3]
E frame #44: PyObject_Call + 0x3e (0x4a587e in /usr/bin/python2)
E frame #45: PyEval_EvalFrameEx + 0x263e (0x4be51e in /usr/bin/python2)
E frame #46: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
E frame #47: PyEval_EvalFrameEx + 0x6076 (0x4c1f56 in /usr/bin/python2)
E frame #48: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
E frame #49: PyEval_EvalFrameEx + 0x58e6 (0x4c17c6 in /usr/bin/python2)
E frame #50: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
E frame #51: /usr/bin/python2() [0x4d57a3]
E frame #52: PyObject_Call + 0x3e (0x4a587e in /usr/bin/python2)
E frame #53: PyEval_EvalFrameEx + 0x263e (0x4be51e in /usr/bin/python2)
E frame #54: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
E frame #55: PyEval_EvalFrameEx + 0x58e6 (0x4c17c6 in /usr/bin/python2)
E frame #56: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
E frame #57: PyEval_EvalFrameEx + 0x58e6 (0x4c17c6 in /usr/bin/python2)
E frame #58: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
E frame #59: /usr/bin/python2() [0x4d5669]
E frame #60: PyObject_Call + 0x3e (0x4a587e in /usr/bin/python2)
E frame #61: PyEval_EvalFrameEx + 0x263e (0x4be51e in /usr/bin/python2)
E frame #62: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
E frame #63: PyEval_EvalFrameEx + 0x58e6 (0x4c17c6 in /usr/bin/python2)
/root/.local/lib/python2.7/site-packages/caffe2/python/workspace.py:211: RuntimeError
----------------------------- Captured stderr call -----------------------------
WARNING:caffe2.python.workspace:Original python traceback for operator 2 in network test_numa in exception above (most recent call last):
WARNING:caffe2.python.workspace: File “/usr/lib/python2.7/runpy.py”, line 174, in _run_module_as_main
WARNING:caffe2.python.workspace: File “/usr/lib/python2.7/runpy.py”, line 72, in _run_code
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/pytest.py”, line 101, in
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/_pytest/config/init.py”, line 79, in main
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/pluggy/hooks.py”, line 289, in call
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/pluggy/manager.py”, line 68, in _hookexec
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/pluggy/manager.py”, line 62, in
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/pluggy/callers.py”, line 187, in _multicall
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/_pytest/main.py”, line 242, in pytest_cmdline_main
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/_pytest/main.py”, line 205, in wrap_session
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/_pytest/main.py”, line 249, in _main
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/pluggy/hooks.py”, line 289, in call
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/pluggy/manager.py”, line 68, in _hookexec
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/pluggy/manager.py”, line 62, in
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/pluggy/callers.py”, line 187, in _multicall
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/_pytest/main.py”, line 270, in pytest_runtestloop
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/pluggy/hooks.py”, line 289, in call
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/pluggy/manager.py”, line 68, in _hookexec
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/pluggy/manager.py”, line 62, in
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/pluggy/callers.py”, line 187, in _multicall
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/_pytest/runner.py”, line 77, in pytest_runtest_protocol
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/_pytest/runner.py”, line 92, in runtestprotocol
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/_pytest/runner.py”, line 172, in call_and_report
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/_pytest/runner.py”, line 197, in call_runtest_hook
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/_pytest/runner.py”, line 225, in from_call
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/_pytest/runner.py”, line 197, in
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/pluggy/hooks.py”, line 289, in call
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/pluggy/manager.py”, line 68, in _hookexec
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/pluggy/manager.py”, line 62, in
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/pluggy/callers.py”, line 187, in _multicall
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/_pytest/runner.py”, line 122, in pytest_runtest_call
WARNING:caffe2.python.workspace: File “/usr/local/lib/python2.7/dist-packages/_pytest/unittest.py”, line 221, in runtest
WARNING:caffe2.python.workspace: File “/usr/lib/python2.7/unittest/case.py”, line 393, in call
WARNING:caffe2.python.workspace: File “/usr/lib/python2.7/unittest/case.py”, line 329, in run
WARNING:caffe2.python.workspace: File “/root/.local/lib/python2.7/site-packages/caffe2/python/numa_test.py”, line 44, in test_numa
WARNING:caffe2.python.workspace: File “/root/.local/lib/python2.7/site-packages/caffe2/python/numa_test.py”, line 32, in build_test_net
------------------------------ Captured log call -------------------------------
WARNING caffe2.python.workspace:workspace.py:218 Original python traceback for operator 2 in network test_numa in exception above (most recent call last):
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/lib/python2.7/runpy.py”, line 174, in _run_module_as_main
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/lib/python2.7/runpy.py”, line 72, in _run_code
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/pytest.py”, line 101, in
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/_pytest/config/init.py”, line 79, in main
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/pluggy/hooks.py”, line 289, in call
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/pluggy/manager.py”, line 68, in _hookexec
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/pluggy/manager.py”, line 62, in
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/pluggy/callers.py”, line 187, in _multicall
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/_pytest/main.py”, line 242, in pytest_cmdline_main
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/_pytest/main.py”, line 205, in wrap_session
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/_pytest/main.py”, line 249, in _main
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/pluggy/hooks.py”, line 289, in call
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/pluggy/manager.py”, line 68, in _hookexec
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/pluggy/manager.py”, line 62, in
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/pluggy/callers.py”, line 187, in _multicall
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/_pytest/main.py”, line 270, in pytest_runtestloop
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/pluggy/hooks.py”, line 289, in call
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/pluggy/manager.py”, line 68, in _hookexec
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/pluggy/manager.py”, line 62, in
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/pluggy/callers.py”, line 187, in _multicall
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/_pytest/runner.py”, line 77, in pytest_runtest_protocol
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/_pytest/runner.py”, line 92, in runtestprotocol
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/_pytest/runner.py”, line 172, in call_and_report
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/_pytest/runner.py”, line 197, in call_runtest_hook
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/_pytest/runner.py”, line 225, in from_call
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/_pytest/runner.py”, line 197, in
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/pluggy/hooks.py”, line 289, in call
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/pluggy/manager.py”, line 68, in _hookexec
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/pluggy/manager.py”, line 62, in
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/pluggy/callers.py”, line 187, in _multicall
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/_pytest/runner.py”, line 122, in pytest_runtest_call
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/local/lib/python2.7/dist-packages/_pytest/unittest.py”, line 221, in runtest
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/lib/python2.7/unittest/case.py”, line 393, in call
WARNING caffe2.python.workspace:workspace.py:223 File “/usr/lib/python2.7/unittest/case.py”, line 329, in run
WARNING caffe2.python.workspace:workspace.py:223 File “/root/.local/lib/python2.7/site-packages/caffe2/python/numa_test.py”, line 44, in test_numa
WARNING caffe2.python.workspace:workspace.py:223 File “/root/.local/lib/python2.7/site-packages/caffe2/python/numa_test.py”, line 32, in build_test_net
------- generated xml file: /home/caffe2/caffe2_tests/python/result.xml --------
---------- onnx coverage: ----------
Operators (passed/loaded/total): 0/0/155
Operator Attributes
(name: #values)======= 1 failed, 392 passed, 7 skipped, 251 warnings in 492.91 seconds ========